GCP ã¢ãœã·ãšã€ãããŒã¿ãã©ã¯ãã£ã·ã§ã㌠(ADP)ïŒæ°ãããšã³ããªãŒã¬ãã«ã®ããŒã¿èªå®è³æ Œ
ADP 㯠2024 幎åŸåã«ãPDE ã®äžäœã«äœçœ®ããã¢ãœã·ãšã€ãã¬ãã«ã® GCP ããŒã¿èªå®è³æ ŒãšããŠç»å ŽããŸããããã®å 容ãšãã©ã®ãããªäººãååŸãã¹ããã玹ä»ããŸãã
é·å¹ŽãGCP ã®ããŒã¿èªå®ãã¹ã«ã¯ç©ºçœããããŸãããCDLïŒåºç€çãæŠå¿µçãSQLãªãïŒãååŸãããããããã¯ã·ã§ã³ããŒã¿ã®å®åçµéšãæ°å¹Žå¿ èŠã§ãDataflow ã®å éšãã¹ããŒãèšèšãããŒã¿ãªãœãŒã¹ã«å¯Ÿãã IAM ãªã©ãæºåäžè¶³ã®åéšè ã«ã¯éåžžã«é£ããã¬ãã«ã«èžã¿èŸŒã Professional Data Engineer (PDE) ã«çŽè¡ãããã®ã©ã¡ããã§ããããã®äžéã¯ç©ºã£ãœã ã£ãã®ã§ãã
Google 㯠2024 幎åŸåã«ãAssociate Data Practitioner (ADP) ã§ãã®ç©ºçœãåããŸããããã㯠125 ãã«ã®ã¢ãœã·ãšã€ãã¬ãã«ã®è©Šéšã§ãçŽ 50 åã2 æéã®æ§æã§ãGCP ã§ã®ããŒã¿ãã£ãªã¢ã 1 ïœ 2 幎ã®äººã ãå ·äœçã«ã¯ãžã¥ãã¢ããŒã¿ãšã³ãžãã¢ãã¢ããªã¹ããšã³ãžãã¢ããã€ãã©ã€ã³æ§ç¯ã«é¢å¿ã®ãã BI æ åœè ããã㊠GCP ã¹ã¿ãã¯ãåŠç¿äžã® AWS / Azure ããŒã¿æ åœè ãæç¢ºã«å¯Ÿè±¡ãšããŠããŸãã
ADPãå®éã«ãã¹ãããå 容
詊éšã®ç®ç㯠2024 幎 10 æã«å ¬éããã2025 幎åã°ã« minor revision ãå ããããŸããã5 ã€ã®ãã¡ã€ã³ããããŸãã
- ããŒã¿åã蟌ã¿ãšåŠçã ã©ã³ãã£ã³ã°ãŸãŒã³ãšããŠã® Cloud StorageãBigQuery ã®ããŒãïŒãããããã³ã¹ããªãŒãã³ã°ïŒãã€ãã³ãåã蟌ã¿ã®ããã® Pub/SubãDataflow ãã³ãã¬ãŒãïŒDataflow ã®å éšã§ã¯ãªãããã㯠PDEïŒãCDC ã®ããã® Datastreamã
- ããŒã¿ã¹ãã¬ãŒãžãšã¢ããªã³ã°ã BigQuery ã®ããŒã¿ã»ãããããŒãã£ã·ã§ãã³ã°ãã¯ã©ã¹ã¿ãªã³ã°ããããªã¢ã©ã€ãºããã¥ãŒãããŒãã«ã¿ã€ãïŒãããŒãžããå€éšãBigLakeïŒãCloud Storage ã®ã¯ã©ã¹ãç¹å®ã®ã¯ãŒã¯ããŒãã«å¿ãã BigQuery / Cloud SQL / Firestore / Bigtable ã®éžæã
- ããŒã¿ãªãŒã±ã¹ãã¬ãŒã·ã§ã³ã SQL 倿ã®ããã® DataformïŒGCP äžã®ææ°ã® dbt çžåœå - Google 㯠2020 幎ã«è²·åããçŸåšã¯ãã¡ãŒã¹ãããŒãã£è£œåïŒãåºæ¬ç㪠Cloud ComposerïŒãããŒãžã AirflowïŒãCloud Schedulerã
- ããŒã¿åæãšå¯èŠåã BigQuery SQL - å®éã® SQLããŠã£ã³ããŠé¢æ°ãçµåãBigQuery ç¬èªã®é å / æ§é äœãå«ã - Looker StudioãLookerïŒã¯ããåé¡ãã³ã¯ã¯ããããåºå¥ããŠãããã¯ããããã¯çŽããããã§ãïŒãåºæ¬ç㪠Connected Sheetsã
- éçšãç£èŠãã»ãã¥ãªãã£ã ããŒã¿ãªãœãŒã¹ã«å¯Ÿãã IAMïŒããŒã¿ã»ããã¬ãã« vs ããŒãã«ã¬ãã« vs è¡ã¬ãã« vs åã¬ãã«ã®ã¢ã¯ã»ã¹ïŒãCloud DLPãåºæ¬ç㪠Dataplex ã¬ããã³ã¹ãBigQuery ã¯ãšãªãã©ã³ / EXPLAIN ãçšããã¯ãšãªæé©åã
ãŸãããã©ã®ã¯ãŒã¯ããŒãã«ã©ã®ãµãŒãã¹ãããšããæææ±ºå®ã«é¢ãã質åãããªãã®å²åãå ããŠããŸããããã¯ã¢ãœã·ãšã€ãã¬ãã«ã®ããŒã¿æ åœè ãæ±ããšæåŸ ãããçš®é¡ã®è³ªåã§ãã
ADPãšPDEã®æ¯èŒ
| ADP | PDE | |
|---|---|---|
| ã¬ãã« | ã¢ãœã·ãšã€ã | ãããã§ãã·ã§ãã« |
| è²»çš | $125 | $200 |
| æé | 2æéãçŽ50å | 2æéãçŽ50å |
| æå¹æé | 3幎 | 2幎 |
| å¿ èŠçµéšå¹Žæ° | GCPããŒã¿çŽ1幎 | æ¥ççŽ3幎ãGCPçŽ1å¹Žä»¥äž |
| Dataflow ã®æ·±ã | ãã³ãã¬ãŒããšåºæ¬æŠå¿µ | ã«ã¹ã¿ã ãã€ãã©ã€ã³ããŠã£ã³ããŠåŠçãé å»¶ããŒã¿ãexactly-once ã»ãã³ãã£ã¯ã¹ |
| BigQuery ã®æ·±ã | ããŒãã£ã·ã§ãã³ã°ãã¯ã©ã¹ã¿ãªã³ã°ãåºæ¬çãªæé©å | ãã£ãã·ãã£ãã©ã³ãã³ã°ãBI Engineãã¹ãããäºçŽãã¯ãšãªãã©ã³ã®è©³çŽ°åæ |
| ã·ããªãªã®è€éã | ã·ã³ã°ã«ãã€ãã©ã€ã³ãã·ã³ã°ã«ãã¡ã€ã³ | ãã«ããã€ãã©ã€ã³ããã«ããã¡ã€ã³ãã³ã¹ã/SLA/ã³ã³ãã©ã€ã¢ã³ã¹å¶çŽä»ã |
ADP 㯠PDE ãžã®æç¢ºãªè¶³ããããšãªããŸããç®æšãååã«éè€ããŠãããããæºåäœæ¥ãç©ã¿éãªã£ãŠå¹æãçºæ®ããŸããããããã·ãã¢ããŒã¿ãšã³ãžãã¢ãªã³ã°ã®åœ¹å²ãç®æããŠããã®ã§ããã°ãPDE ã®ä»£ããã«ã¯ãªããŸãããæ¡çšæ åœè ããã®éããçè§£ããŠããŸããlevels.fyi ã®ããŒã¿ã«ãããšãPDE ã¿ã°ä»ãã®åœ¹å²ã¯ãADP ã¿ã°ä»ãã®åœ¹å²ãããäžäœã«äœçœ®ä»ããããŠããŸãïŒäž¡æ¹ãæ±äººã«åºãããšã¯çšã§ãã - ADP ã¯ãŸã æ°ããããŠããããã«ãã£ã«ã¿ãªã³ã°ã§ããŸããïŒã
ADPãååŸãã¹ã人
ãžã¥ãã¢ããŒã¿ãšã³ãžãã¢ãšã¢ããªã¹ããšã³ãžãã¢ã§ãGCP çµéšã 2 å¹Žæªæºã®äººãADP ã¯ããªãã®ã¬ãã«ã«åã£ãé©åãªè³æ Œã§ãã1 幎ç®ã§ PDE ã«æãåºãã®ã¯ãããŸãããããã®è©Šéšã¯ãå®éã«æ¥åãçµéšããŠããªã人ã«ã¯å³ãããã®ãšãªãããã«äœãããŠããŸãã
ãã€ãã©ã€ã³æ§ç¯ã«ç§»è¡ããã¢ããªã¹ããLooker ã§ SQL ãèšè¿°ãããªãŒã±ã¹ãã¬ãŒã·ã§ã³ããã³åã蟌ã¿å±€ãæ åœãå§ããã°ããã§ããã°ãADP ã¯ãŸãã«å¿ èŠãªäœç³»çãªã«ãªãã¥ã©ã ã§ããDataform + BigQuery + Composer ã®ãã©ã€ã¢ãã¯ãçŸä»£ã® GCP ã¢ããªã¹ããšã³ãžãã¢ã®ã¹ã¿ãã¯ã§ãã
GCP ãåŠç¿äžã® AWS / Azure ããŒã¿ãšã³ãžãã¢ãdbt + Snowflake / Redshift / Synapse ããã§ã«ç¥ã£ãŠããå ŽåãADP ã¯ãã®ã¡ã³ã¿ã«ã¢ãã«ã BigQuery + Dataform + Looker ã«ãããã³ã°ããããã®æéã®ãã¹ã§ãã2 ïœ 3 é±éã®éäžçãªåŠç¿ã§ãçç£çã«ãªããã§ãããã
éããŒã¿ãšã³ãžãã¢ãªã³ã°ããã®ãã£ãªã¢ãã§ã³ãžã£ãŒãããã¯ãšã³ããšã³ãžãã¢ã§ããŒã¿åéã«è»¢åããããšããŠããå ŽåãADP 㯠PDE ã®æ·±ãã§åæ£ã·ã¹ãã ã®å éšãåŠã¶ããšã匷å¶ããããšãªããGCP åºæã®ããŒã¿çšèªãç¿åŸã§ããŸãã
ADPãã¹ããããã¹ã人
GCP ããŒã¿æ¥åã 3 幎以äžè¡ã£ãŠãããDataflow ã®ã«ã¹ã¿ã ãã€ãã©ã€ã³ãã¹ãããäºçŽãçµç¹ã¬ãã«ã§ã® BigQuery ã®ã³ã¹ãæé©åã«ãã§ã«æ £ããŠããã®ã§ããã°ãADP ãã¹ãããããŠçŽæ¥ PDE ã«é²ã¿ãŸããããADP ã¯ãPDE ãæäŸããªããã®ãã·ãã¢ã®å±¥æŽæžã«è¿œå ããããšã¯ãããŸããã
SQL ãããŸã«æžãããã©ããŒã¿ã€ã³ãã©ã管çããŠããªããœãããŠã§ã¢ãšã³ãžãã¢ã§ããã°ãããŒã¿èªå®è³æ Œã¯å¿ èŠãããŸãããACE ãŸã㯠PCA ã GCP ã®åºç€ãã«ããŒããŠãããããªãã®ããŒã¿æ¥åã«ã¯å¥ã®ã·ã°ãã«ã¯å¿ èŠãããŸããã
6é±éã®æºåèšç»
é± 8 æéã®åŠç¿ãšãããŒã¿æ¥åã«éå®ãããªããšã 1 幎éã® GCP çµéšãããããšãåæãšããŸãã
Week 1: BigQuery ã®åºç€ã ããŒã¿ã»ãããããŒãã«ïŒãããŒãžã / å€éš / BigLakeïŒãããŒãã£ã·ã§ãã³ã°ãšã¯ã©ã¹ã¿ãªã³ã°ãBigQuery ã¯ãšãªãã©ã³ãã¹ãããã¢ãã«ã®åºæ¬ããªã³ããã³ã vs ãã£ãã·ãã£èª²éãæŒç¿ïŒå ¬éããŒã¿ã»ãããããŒãããè€éãªã¯ãšãªãæžããã¯ãšãªãã©ã³ã確èªããããŒãã£ã·ã§ãã³ã°ã远å ããŠåèšæž¬ããŸãã
Week 2: åã蟌ã¿ãã¿ãŒã³ã ã€ãã³ãã¹ããªãŒãã³ã°ã®ããã® Pub/Subãããã / ã¹ããªãŒãã³ã° ETL ã®ããã® Dataflow ãã³ãã¬ãŒããããŒã¿ããŒã¹ CDC ã®ããã® DatastreamãBigQuery ã¹ããªãŒãã³ã°æ¿å ¥ãStorage Transfer Serviceããšã³ãããŒãšã³ãã®ãã€ãã©ã€ã³ãæ§ç¯ããŸãïŒPub/Sub â Dataflow ãã³ãã¬ãŒã â BigQueryãCloud Storage ãã¹ããŒãžã³ã°ãã±ãããšããŠäœ¿çšã
Week 3: 倿ãšãªãŒã±ã¹ãã¬ãŒã·ã§ã³ã Dataform - ãã®èªå®è³æ Œãæšé²ããPDE ãéèŠããªãäž»èŠãªãã®ã§ããSQLXãå®çŸ©ãã¢ãµãŒã·ã§ã³ãäŸåé¢ä¿ãã¹ã±ãžã¥ãŒã«ããããªãªãŒã¹ãé SQL ãªãŒã±ã¹ãã¬ãŒã·ã§ã³ã®ããã® Cloud Composerãé± 1 ã§ããŒãããããŒã¿ã«å¯Ÿã㊠Dataform ãããžã§ã¯ããæ§ç¯ããŸãã
Week 4: åæãšå¯èŠåã Looker StudioïŒç¡æãããã·ã¥ããŒãäžå¿ïŒvs LookerïŒææãã»ãã³ãã£ãã¯ã¬ã€ã€ãŒ + LookMLïŒãConnected Sheetsãé«éåãããããã·ã¥ããŒãã®ããã® BigQuery BI Engineãé± 3 ã§è¡ã£ã倿ã®äžã« Looker Studio ããã·ã¥ããŒããæ§ç¯ããŸãã
Week 5: ã¬ããã³ã¹ãšéçšã ããŒã¿ã»ãã / ããŒãã« / è¡ / åã¬ãã«ã® IAMãæ¿èªæžã¿ãã¥ãŒãPII ã®æ€åºãšãã¹ãã³ã°ã®ããã® Cloud DLPãDataplexïŒããŒã¿ãã¡ããªã㯠/ ã¬ããã³ã¹ïŒãç£æ»ãã®ã³ã°ãBigQuery information_schema ãã¥ãŒã«ããã³ã¹ãç£èŠããã®é±ã¯ããã¥ã¡ã³ãã®èªã¿èŸŒã¿ãäžå¿ã§ãã
Week 6: æš¡æ¬è©Šéšã 3 ïœ 5 åã®å®å šãªæéå¶éä»ãå®è¡ãCertLabPro ã®åé¡éãWhizlabsããããŠå ¬åŒã® Cloud Skills Boost æš¡æ¬è©ŠéšãåéšäºçŽåã« 80% 以äžã®ã¹ã³ã¢ãç®æããŸãããã
絊äžã·ã°ãã«
ADP ã¯æ°ããããŠãæ£ç¢ºãªçµŠäžããŒã¿ã¯ãããŸãããæãè¿ã代çã¯ãäž»èŠãªç±³åœã®éœåžåã«ããã BigQuery / Looker ã®ã¢ããªã¹ããšã³ãžãã¢ã®åœ¹å²ã§ãåºæ¬çµŠã¯ 10 äžãã«ãã 14 äžãã«ã§ãïŒlevels.fyi 2025-2026 ã®ã¢ããªãã£ã¯ã¹ãšã³ãžãã¢ããŒã¿ãBuilt In ã®ç¯å²ãBLS OEWS 15-1242 ããŒã¿ããŒã¹ç®¡çè ããã³ã¢ãŒããã¯ãã«ããåºç¯ãªåž¯åïŒããã®åœ¹å²ã« ADP ãå ããããšã§ã瀟å ã§ 5 åãã«ãã 1 äžãã«ã»ã©çµŠäžãäžããå¯èœæ§ããããŸãããã倧ããªå€åã¯ããã®èªå®è³æ ŒãããŒã¿ãšã³ãžãã¢ãªã³ã°ãžã®æ¬æ Œçãªç§»è¡ãå©ããåºæ¬çµŠã 13 äžãã«ãã 18 äžãã«ã«äžæããããšã«ãããŸããPDE ã¯ãã®æ°åãããã«åãããŸããã12 ïœ 18 ãæã§ ADP ã®åŸã« PDE ãååŸãããšããã®ã劥åœãªè»éã§ãããçµéšãè£ä»ããªãã« PDE ãæ¥ããããæ£çŽãªéã§ãããã
çµè«
ADP 㯠GCP ããŒã¿ãã¹ã®çã®ã®ã£ãããåããŸããããªãã GCP ã®ããŒã¿é¢é£ã®è·åã§ãžã¥ãã¢ããããã«ã¬ãã«ã§ããã°ã2026 幎ã«ååŸãã¹ãèªå®è³æ Œã§ããå ¬æ£ãªè©Šéšã§ãããæºåè³æãæç¢ºã§ãè³æ Œãåãªã顿ã§ã¯ãªãå®éã®è·åã¬ãã«ã«ãããã³ã°ãããŸãã
CertLabPro ã§ ADP ã®åé¡éãé²èЧããæºåãã§ãããããŸãã¯ãã§ã«æºåãã§ããŠãããªãæéå¶éä»ã詊éšãéå§ããããšãã§ããŸããæ¬¡ã« PDE ãç®æããŠããŸããïŒ PDE ã®åé¡éã¯ãã¡ãã