Mentranskode data mainframe dari jarak jauh di Google Cloud
Tetap teratur dengan koleksi
Simpan dan kategorikan konten berdasarkan preferensi Anda.
Mentranskode data secara lokal di mainframe adalah proses yang intensif CPU yang menghasilkan konsumsi million instructions per second (MIPS) yang tinggi. Untuk menghindari hal ini, Anda
dapat menggunakan Cloud Run untuk memindahkan dan mentranskode data mainframe dari jarak jauh di
Google Cloud ke format kolom baris yang dioptimalkan (ORC), lalu memindahkan data ke
Cloud Storage. Hal ini akan membebaskan mainframe Anda untuk tugas penting bisnis dan
juga mengurangi konsumsi MIPS.
Gambar berikut menjelaskan cara memindahkan data mainframe keGoogle Cloud dan mentranskodenya dari jarak jauh ke format ORC menggunakan Cloud Run, lalu memindahkan konten ke BigQuery.
Buat akun layanan atau identifikasi akun layanan yang ada untuk digunakan dengan Mainframe Connector. Akun layanan ini harus memiliki izin untuk mengakses bucket Cloud Storage, set data BigQuery, dan resource Google Cloud lainnya yang ingin Anda gunakan.
Pindahkan data mainframe ke Google Cloud dan transkode dari jarak jauh menggunakan Cloud Run
Untuk memindahkan data mainframe ke Google Cloud dan mentranskodenya dari jarak jauh menggunakan
Cloud Run, Anda harus melakukan tugas berikut:
Baca dan transkode set data di mainframe, lalu upload ke Cloud Storage dalam format ORC. Transcoding dilakukan selama operasi gsutil cp, dengan set data kode pertukaran desimal (EBCDIC) kode biner mainframe yang diperluas dikonversi ke format ORC dalam UTF-8 selama penyalinan ke bucket Cloud Storage.
Muat set data ke tabel BigQuery.
(Opsional) Jalankan kueri SQL pada tabel BigQuery.
(Opsional) Mengekspor data dari BigQuery ke file biner di Cloud Storage.
Untuk melakukan tugas ini, ikuti langkah-langkah berikut:
Di mainframe, buat tugas untuk membaca set data di mainframe dan
transcode ke format ORC, sebagai berikut. Baca data dari
set data INFILE, dan
tata letak data dari COPYBOOK DD.
Set data input harus berupa file metode akses berurutan (QSAM) dalam antrean dengan
panjang data tetap atau variabel.
Untuk mengetahui daftar lengkap variabel lingkungan yang didukung oleh
Mainframe Connector, lihat Variabel lingkungan.
(Opsional) Buat dan kirim tugas kueri BigQuery yang menjalankan pembacaan SQL dari file DD QUERY.
Biasanya, kueri akan berupa pernyataan MERGE atau SELECT INTO DML
yang menghasilkan transformasi tabel BigQuery. Perhatikan bahwa Konektor Mainframe mencatat metrik tugas, tetapi tidak menulis hasil kueri ke file.
Anda dapat membuat kueri BigQuery dengan berbagai cara-inline, dengan set data terpisah menggunakan DD, atau dengan set data terpisah menggunakan DSN.
Selain itu, Anda harus menetapkan variabel lingkungan BQ_QUERY_REMOTE_EXECUTION=true.
Ganti kode berikut:
PROJECT_NAME: Nama project tempat Anda
ingin menjalankan kueri.
LOCATION: Lokasi tempat kueri akan
dijalankan. Sebaiknya jalankan kueri di lokasi yang dekat dengan data.
(Opsional) Buat dan kirim tugas ekspor yang mengeksekusi pembacaan SQL dari
file DD QUERY, dan ekspor
set data yang dihasilkan ke Cloud Storage sebagai file biner.
[[["Mudah dipahami","easyToUnderstand","thumb-up"],["Memecahkan masalah saya","solvedMyProblem","thumb-up"],["Lainnya","otherUp","thumb-up"]],[["Sulit dipahami","hardToUnderstand","thumb-down"],["Informasi atau kode contoh salah","incorrectInformationOrSampleCode","thumb-down"],["Informasi/contoh yang saya butuhkan tidak ada","missingTheInformationSamplesINeed","thumb-down"],["Masalah terjemahan","translationIssue","thumb-down"],["Lainnya","otherDown","thumb-down"]],["Terakhir diperbarui pada 2025-08-19 UTC."],[],[],null,["# Transcode mainframe data remotely on Google Cloud\n\nTranscoding data locally on a mainframe is a CPU-intensive process that results\nin high million instructions per second (MIPS) consumption. To avoid this, you\ncan use Cloud Run to move and transcode mainframe data remotely on\nGoogle Cloud to optimized row columnar (ORC) format and then move the data to\nCloud Storage. This frees up your mainframe for business critical tasks and\nalso reduces MIPS consumption.\n\nThe following figure describes how you can move your mainframe data to\nGoogle Cloud and transcode it remotely to ORC format using\nCloud Run, and then move the content to BigQuery.\n\n\u003cbr /\u003e\n\nRemotely transcode mainframe data\n\n\u003cbr /\u003e\n\nBefore you begin\n----------------\n\n- [Deploy Mainframe Connector on Cloud Run](/mainframe-connector/docs/deploy-mainframe-connector).\n- [Create a service account](/iam/docs/service-accounts-create) or identify an existing service account to use with Mainframe Connector. This service account must have permissions to access Cloud Storage buckets, BigQuery datasets, and any other Google Cloud resource that you want to use.\n- Verify that the service account you created is assigned the [Cloud Run Invoker role](/run/docs/reference/iam/roles#run.invoker).\n\nMove mainframe data to Google Cloud and transcode it remotely using Cloud Run\n-----------------------------------------------------------------------------\n\nTo move your mainframe data to Google Cloud and transcode it remotely using\nCloud Run, you must perform the following tasks:\n\n1. Read and transcode a dataset on a mainframe, and upload it to Cloud Storage in ORC format. Transcoding is done during the [`gsutil cp`](/mainframe-connector/docs/api-command-reference#gsutil_cp) operation, where a mainframe extended binary coded decimal interchange code (EBCDIC) dataset is converted to the ORC format in UTF-8 during the copy to a Cloud Storage bucket.\n2. Load the dataset to a BigQuery table.\n3. (Optional) Execute a SQL query on the BigQuery table.\n4. (Optional) Export data from BigQuery into a binary file in Cloud Storage.\n\nTo perform these tasks, follow these steps:\n\n1. On your mainframe, create a job to read the dataset on your mainframe and\n transcode it to ORC format, as follows. Read the data from the\n [INFILE dataset](/mainframe-connector/docs/reference#dataset-names), and the\n record layout from the [COPYBOOK DD](/mainframe-connector/docs/reference#dataset-names).\n The input dataset must be a queued sequential access method (QSAM) file with\n fixed or variable record length.\n\n | **Note**\n | - Not all Google Cloud commands support remote transcoding. For more information, see [Mainframe Connector API reference](/mainframe-connector/docs/reference).\n | - Variables with the suffix `FILLER` are ignored during the import process.\n | - From version 5.12.0 onwards, Mainframe Connector replaces hyphens (\"-\") with underscores (\"_\") in variable names. If you want to keep hyphens in your variable names, disable this automatic conversion by setting the database variable `BQSH_FEATURE_CONVERT_UNDERSCORE_IN_FIELDS_NAME` to `false`.\n\n For the complete list of environment variables supported by\n Mainframe Connector, see [Environment variables](/mainframe-connector/docs/environment-variables). \n\n //STEP01 EXEC BQSH\n //INFILE DD DSN=\u003cHLQ\u003e.DATA.FILENAME,DISP=SHR\n //COPYBOOK DD DISP=SHR,DSN=\u003cHLQ\u003e.COPYBOOK.FILENAME\n //STDIN DD *\n gsutil cp --replace gs://mybucket/tablename.orc --remote \\\n --remoteHost \u003cmainframe-connector-url\u003e.a.run.app \\\n --remotePort 443\n /*\n\n If you want to log the commands executed during this process, you can [enable load statistics](/mainframe-connector/docs/reference#enable_load_statistics).\n2. (Optional) Create and submit a BigQuery query job that executes a SQL read from\n the [QUERY DD file](/mainframe-connector/docs/reference#dataset-names).\n Typically the query will be a `MERGE` or `SELECT INTO DML`\n statement that results in transformation of a BigQuery table. Note\n that Mainframe Connector logs in job metrics but doesn't write query\n results to a file.\n\n You can query BigQuery in various ways-inline, with a separate\n dataset using DD, or with a separate dataset using DSN. \n\n Example JCL\n //STEP03 EXEC BQSH\n //QUERY DD DSN=\u003cHLQ\u003e.QUERY.FILENAME,DISP=SHR\n //STDIN DD *\n PROJECT=\u003cvar translate=\"no\"\u003ePROJECT_NAME\u003c/var\u003e\n LOCATION=\u003cvar translate=\"no\"\u003eLOCATION\u003c/var\u003e\n bq query --project_id=$PROJECT \\\n --location=$LOCATION \\\n --remoteHost \u003cmainframe-connector-url\u003e.a.run.app \\\n --remotePort 443/*\n /*\n\n Additionally, you must set the environment variable `BQ_QUERY_REMOTE_EXECUTION=true`.\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003ePROJECT_NAME\u003c/var\u003e: The name of the project in which you want to execute the query.\n - \u003cvar translate=\"no\"\u003eLOCATION\u003c/var\u003e: The location for where the query will be executed. We recommended that you execute the query in a location close to the data.\n3. (Optional) Create and submit an export job that executes a SQL read from the\n [QUERY DD file](/mainframe-connector/docs/reference#dataset-names), and exports\n the resulting dataset to Cloud Storage as a binary file.\n\n Example JCL\n //STEP04 EXEC BQSH\n //OUTFILE DD DSN=\u003cHLQ\u003e.DATA.FILENAME,DISP=SHR\n //COPYBOOK DD DISP=SHR,DSN=\u003cHLQ\u003e.COPYBOOK.FILENAME\n //QUERY DD DSN=\u003cHLQ\u003e.QUERY.FILENAME,DISP=SHR\n //STDIN DD *\n PROJECT=\u003cvar translate=\"no\"\u003ePROJECT_NAME\u003c/var\u003e\n DATASET_ID=\u003cvar translate=\"no\"\u003eDATASET_ID\u003c/var\u003e\n DESTINATION_TABLE=\u003cvar translate=\"no\"\u003eDESTINATION_TABLE\u003c/var\u003e\n BUCKET=\u003cvar translate=\"no\"\u003eBUCKET\u003c/var\u003e\n bq export --project_id=$PROJECT \\\n --dataset_id=$DATASET_ID \\\n --destination_table=$DESTINATION_TABLE \\\n --location=\"US\" \\\n --bucket=$BUCKET \\\n --remoteHost \u003cmainframe-connector-url\u003e.a.run.app \\\n --remotePort 443\n /*\n\n Replace the following:\n - \u003cvar translate=\"no\"\u003ePROJECT_NAME\u003c/var\u003e: The name of the project in which you want to execute the query.\n - \u003cvar translate=\"no\"\u003eDATASET_ID\u003c/var\u003e: The BigQuery dataset ID that contains the table that you want to export.\n - \u003cvar translate=\"no\"\u003eDESTINATION_TABLE\u003c/var\u003e: The BigQuery table that you want to export.\n - \u003cvar translate=\"no\"\u003eBUCKET\u003c/var\u003e: The Cloud Storage bucket that will contain the output binary file.\n\nWhat's next\n-----------\n\n- [Move locally transcoded mainframe data to Google Cloud](/mainframe-connector/docs/local-transcoding)\n- [Transcode mainframe data remotely on Google Cloud](/mainframe-connector/docs/remote-transcoding)\n- [Transcode mainframe data moved to Google Cloud using a virtual tape library](/mainframe-connector/docs/vtl-transcoding)"]]
