A discussion between a CxO and a senior Data Architect Part 3

A discussion between a CxO and a senior Data Architect Part 3

Links to other parts

A discussion between a CxO and a senior Data Architect Part 1

A discussion between a CxO and a senior Data Architect Part 2

A discussion between a CxO and a senior Data Architect Part 4

A discussion between a CxO and a senior Data Architect Part 5

.

Background: We have been going through a discussion that took place between senior leadership and a data architect. They took 45 min of lunch break and here Part – 3 continues.

.

Discussion Follows:

.

Alison: Hope you enjoyed your food.

Vasumat: Yup, more than that, I liked that elegant garden, especially the spectacular mountain view from the corner.

Alison: Hmm, yes, that’s a favorite coffee spot for everyone on the floor.

Alison: Coming to the business, now I would like to discuss your area of expertise. I believe you have extensive experience in migrating data platforms. Isn’t it?

Vasumat: True!

.

Alison: Can you describe the steps involved in migrating database systems from on-premises to the cloud? Well, let me simplify my question, let’s suppose we have some heterogeneous database systems like Microsoft, Oracle, etc. I would like to know how you handle the database migrations. Since we already discussed phases, you can explain the series of steps involved. Appreciate it if you can explain it with whiteboarding. Take your time and explain as much detail as possible.

Vasumat: Sure! Assuming we already have an approved business case and talent is assigned.

.

Migration Planning:

 Assess and Discover: Collect every possible information and prepare a data-estate / data-sheet / metadata about Data (file shares, network shares, data lakes, etc.) and Database systems (SQL, NoSQL, etc.) from the source environment. It includes database instances (RDBMS, versions, edition, license etc.), instance type (database engine, reporting, analytics, operational etc.), configurations (trace flags, collation etc.), features (FileStream, PolyBase etc.), host details (virtualization, physical servers, OS type, license, etc.), computing power (memory, disk, CPU, throughput etc.), storage – data volume (data size, data growth rate etc.), partition info (keys, distribution rate), data type (Transactional, analytical, Archive, warehouse, reporting, Staging etc.), environment (DEV, Testing, QA, Staging, Pre-Prod, Prod), tools (development, deployment, monitoring), networking, security (NSG, firewalls, SSL, encryption, login – passwords, service accounts, permissions etc.), backup strategy, high-availability, disaster recovery, load balancing, data dependencies, Database IT operations (Change request, incident, problem request management etc.), Key Performance Baselines and Indicators (IO, CPU, Memory, Query timeouts, errors ) etc.
 Integration Matrix: A data entity can be a “data provider” “data consumer” or both. We must prepare a matrix that can project all dependencies of our data entity/database. This means identifying the list of applications, services, and programs that are using our databases. Also defining and tagging application with priority (based on the revenue impact), complexity (number of components involved), route (source, destination, IP, port, port type, inbound/outbound, access credentials, etc.) information. It plays a vital role in determining the migration scope.
 Define the Migration Scope: Based on the assessment details, we can classify the data entities between, moving to the cloud, retaining at on-premises, and retiring/decommissioning.
 Define the Cloud solution: Need to identify
 Cloud model: Public, private, Hybrid, or Multi-Cloud
 Cloud Service provider: Azure, AWS, Google Cloud, Oracle Cloud Infrastructure, etc.
 Cloud Service model: Platform-as-a-Service, Infrastructure-as-a-Service, Software-as-a-Service. From the DB side, we deal with PaaS and IaaS. We conclude this in the resource mapping step.
 Resource mapping: Map on-premises resources with the cloud resources. If we plan to re-platform or refactor, check compatibility, and perform a POC (Proof of Concept) in the cloud. Sizing the cloud resources must be handled by considering the expected performance and allocated budget. In general, we allocate a little higher compute resources than required, so it helps us for faster migrations. Once migration is done, we can downgrade the service tier to match the exact required size.
 For example, Map Database systems to Azure SQL Database, Database Pool, Managed Instance, AWS RDS/Aurora, Azure SQL Serverless, AWS Aurora Serverless, Azure VM, AWS EC2, etc.; Servers or Virtual Machines to Azure VM, AWS EC2; Network drives and shared folders to Azure Files, AWS Elastic File System (EFS); Web Applications to Azure App Service, AWS Elastic Beanstalk; Virtual server disks to Azure Managed Disks, AWS Elastic Block Storage (EBS); Storage solutions to Azure Blob Storage, Data Lake, AWS Simple Storage Services (S3); NoSQL to Azure Cosmos DB API’s, AWS DynamoDB, Simple DB, Document DB; Load balancer to Azure Load Balancer, AWS Network Load Balancer; Programs to Elastic Jobs, Azure Functions; Workflows and jobs to Logic Apps; ETL solutions to Data Factory, Databricks, Synapse pipelines, AWS Glue, etc.; Datawarehouse to Azure Synapse Analytics, AWS Redshift, etc.; CI/CD pipelines to Azure DevOps release pipelines, AWS CodePipeline, etc. Data shares to Azure Data Share, AWS Lake Formation, etc.
 Define the cloud migration approach and tools: For each component marked for migration tag it with one of the cloud migration strategies (Rehost, Refactor, re-architect, etc.). From the database aspect mostly we do either rehost (IaaS) or refactor (PaaS). Based on the migration requirement (Online, Offline) we need to select the right migration tool. Ex: Database Migration Service, Data Migration Assistant, Backup & restore, replication, AlwaysOn, export/import, Visual Studio, bulk load, ETL tools () third party tools, etc.
 Define the rollback plan: We need to prepare a Rollback plan and typically for 4 scenarios.
 A) Migrated, application testing failed, rollback.
 B) Migrated, started using the system in the cloud (data changes, schema changes), realized something is not working or failing after a few days, rollback
 C) Rollback from PaaS deployment to On-premises
 D) Rollback from IaaS to On-premises.
 Summary: Scenario A is easy as we simply need to update the connection string back to on-premises. Scenario B & C requires a lot of effort as we need to compare the databases between source and target and identify the changes and apply those changes to the on-premises database. Scenario D will be a little easier compared to B & C options as we can easily perform backup-restore from cloud to on-premises or we can configure HA&DR solution between Cloud VM and on-premises Database Instance.

.

Pre-Migration Activity:

Continue reading

Posted in Interview Q&A | Tagged , , , , , , , , , , , , , , | Leave a comment

A discussion between a CxO and a senior Data Architect Part 2

A discussion between a CxO and a senior Data Architect Part 2

Links to other parts

A discussion between a CxO and a senior Data Architect Part 1

A discussion between a CxO and a senior Data Architect Part 3

A discussion between a CxO and a senior Data Architect Part 4

A discussion between a CxO and a senior Data Architect Part 5

.

Background: We have been going through a discussion that took place between senior leadership and a data architect. They took a coffee break and here Part – 2 continues.

.

<During the break, had some casual talks about personal life and hobbies, etc. which is irrelevant here. We’ll see the discussion after the break>

.

Discussion Follows:

.

Alison: So, Vasumat, let’s say a CxO level employee asked you “why we should migrate to the Cloud?”. Assume that person doesn’t care about technology, but business. You need to convince him/her. Can you?

Vasumat: Well, that’s a reasonable question and we call it a Business Case. That’s what we do in the 0th phase “Migration Preparation and Business Planning”. Before starting the migration project, we need to establish a Business Case for migration, and set expectations for cost, return, and the timing of those costs and returns. Money is everything in business, so the migration business case showcases the TCO (Total Cost of Ownership) and ROI (Return On Investment) calculations that tell the benefit of migrating to the cloud.

Alison: Can you define the terms TCO and ROI?

Vasumat: Sure! Those are the key parameters to establish the business case. TCO and ROI both talk about money we need to pay from our pocket and the gain or return that we get on the investment.

Alison: I am listening….

Vasumat:

 TCO (Total Cost Ownership): is the sum of the expenses associated with purchasing, deploying, using, and retiring a product/equipment/asset. Essentially, the sum of the purchase and operating price of an asset for its lifetime. There are two types of costs involved direct and indirect. Ex: When we buy a car:
 Direct cost: Car retail price, registration tax, road tax, insurance premium, excise duty, state tax, monthly fuel expenses, car maintenance and servicing cost, etc.
 Indirect Cost: Parking tickets and toll charges come under hidden or indirect charges.
 Car TCO: The total of all Direct and Indirect expenses is called the car TCO. Likewise, we need to calculate the overall TCO for our organization’s technology infrastructure which includes both hardware, software, and operations costs.
 On-premises TCO calculation: We usually perform infrastructure audits to identify the direct and indirect costs
 Direct costs:  Direct costs are relatively easy to calculate, as they remain on our organization’s balance sheet. It is categorized into hardware (physical assets), software (licensing, maintenance, upgrades, etc.), management (architect, hosting, reporting, etc.), support (supporting staff), and implementation (development, customizing, and integrating). Ex: Physical servers, storage devices, network devices, printers, software licenses, maintenance contracts, warranties, supplies, material, spare parts, real estate/space, internet facility, labor/resources, hiring, onboarding, training employees, maintenance, upgrades, security, disaster recovery, etc.
 Indirect costs: Difficult to calculate but equally important as direct costs. Ex: Un-scheduled maintenance due to breakdown or failure, business impact due to unexpected downtime, power, and energy consumptions, depreciation of asset performance and cost over time, etc.
 Cloud TCO calculation: Almost all the cloud providers are offering an advanced TCO calculator. Based on our on-premises analysis we can input the infrastructure details (servers, OS, cores, memory, storage, licensing, etc.) and it will automatically calculate and report the estimated cloud TCO. Additionally, we need to consider the one-time investment in migration activity.
 Direct: Cloud services & subscription, integration & testing, consulting & training, and Migration cost. Ex: Database service, App service, VM, Load balancer, data ingestion rates, storage, application testing, training on cloud technology, hiring for cloud migrations, the cost for 6R strategy (ex: upgrading to new versions), etc.
 Indirect: Excess compute and storage reservations, premium service packages with no proper SLA (no guarantee on gains Ex: performance, cost, support), software licenses used or forfeited (loosing or giving up) in the transition, required infrastructure at on-premises after the cloud migration (hybrid model), hosting without POC can cause unexpected cost increments, and user behavior (untrained users may provision a lot of unnecessary services without knowing the consequences from cost point), etc.
 ROI (Return on investment): It tells us how much profit or loss our investment has earned. ROI is the ratio of a profit or loss made in a fiscal year, expressed as a percentage. Here is the simple formula:
 General formula: ROI = (Net Profit / Cost of Investment) x 100
 *Net Profit = Gain from the investment – Cost of Investment

.

 ROI formula commonly used in cloud migrations:
 ROI = ((FVI – IVI) / Cost of Investment) X 100
 *FVI: Final Value of Investment
 *IVI: Initial Value of Investment

Continue reading

Posted in Interview Q&A | Tagged , , , , , , , , , , , | Leave a comment

A discussion between a CxO and a senior Data Architect Part 1

.

A discussion between a CxO and a senior Data Architect Part 1

Links to other parts

A discussion between a CxO and a senior Data Architect Part 2

A discussion between a CxO and a senior Data Architect Part 3

A discussion between a CxO and a senior Data Architect Part 4

A discussion between a CxO and a senior Data Architect Part 5

.

.

Below is the conversation that happened in an interview for the role of Lead Data Architect. It was more like business analysis and technical discussion between a client and a consultant. We try our best to narrate the same way as it happened in the interview. We are changing the names for privacy purposes, assuming that the interviewer is Alison, and the interviewee is Vasumat.

Mr. Vasumat has already gone through 1 coding round and 3 technical discussions within the span of 8 weeks. Now, he is having the final discussion with senior leadership. He was informed that he may need to prepare to spend 6 to 8 hours (including lunchtime) in their office. Vasumat reached the office and completed his entry formalities by 8:45 AM whereas his interview is scheduled for 9:15 AM. HR introduced Vasumat to the interviewer.

.

Here you go:

.

Alison: It’s nice to meet you Mr. Vasumat

Vasumat: Honor is all mine mam

Alison: You can call me Alison.

 I am a chief program director working with XXXXX for the last 20 years and I take care of the digital transformation and modernization of our software applications.
 We have a humongous enterprise that includes thousands of servers, databases, and petabytes of data with heterogeneous database systems. We’ve been planning to migrate our applications to the cloud.
 That’s the reason I’ve been staying here in India for the next few months to settle down the tech stack and for the migration kickoff. We hired cloud architects and looking for a data architect who can handle end-to-end data migration.
 We have a partnership with Microsoft, so Azure will be the primary choice. However, down the line, we would like to go with multi-cloud architecture most probably with Azure and Google.
 Richard told me that you have considerable experience, especially in data platforms and cloud migrations. Also, from the previous discussions, they have a positive outlook on your profile. I just wanted to discuss it before finalizing.
 So Vasumat, I would like to hear about your experience with cloud migrations

.

Vasumat: Thanks for the detailed background. I have over 13 years of experience in dealing with enterprise database systems. My cloud journey started 7 years back and earned extensive knowledge in migrating heterogeneous database systems from on-premises to the cloud. Experience in both Azure and AWS but my primary specialization is in Azure.

Alison: Since you are an architect, you must have participated in the migration planning and execution, isn’t it?

Vasumat: Certainly! It’s part of my role.

.

Alison: Great, can you describe the phases of cloud migration at a high level?

Vasumat: There are 5 (1+4) phases involved:

 Migration Preparation and Business Planning + Discovery, Planning, Execution, and Optimization.

Alison: I am listening…

Vasumat: Let me explain in detail.

 Migration Preparation and Business Planning: We create a business case for migration by projecting benefits in terms of cost, performance, and digitalization aspects and seek business approvals. Before starting the next phase, we must have a dedicated stakeholder (Cloud/Solution/Migration/Data Architect) who can lead and execute the migration project.
 Phase 0 – Output: A clear Business Case that tells us why we are migrating to the cloud

.

 Discovery & Assessment: We understand, analyze, and collect data from the source environment. This includes infrastructure, software, hardware, network & security, operational models, enterprise policies and agreements, and data landscape. I do take care of the data part.
 Software applications: web, desktop, mobile, message broker, web service, etc.
 Application dependencies: Custom libraries, parent and child apps, integrations, etc. Ex: Our web app is using a Currency conversion service; we built a custom library to convert an invoice query result to a PDF etc.
 Legacy applications: Unsupported / Older versions, mainframes, etc.
 Third-party solutions: Collibra, Meta-Center, HVR replication, Commvault, etc.
 Client Tools: Visual Studio, Eclipse, Oracle developer, SSMS, MySQL Workbench, etc.
 Source code repository: TFS, SVN, Git, Bitbucket, etc.
 Operating systems and capacity: Windows Linux, Mac, Memory, CPU, IO, Disks, etc.
 Licensing: Opensource, corporate agreements, renewal status, licensing mobility, etc.
 Hardware mapping: Servers, Racks, Cables, Switches, Storage, Routers, Power Equipment, Colling Systems, Network devices, routers, etc.
 Servers: Physical / Virtual, Application, Web, Database, Proxy, Mail, FTP, Print, DNS, etc.
 Network and Security: SSL, TLS, Encryption, firewalls, network security groups, web application and third-party firewalls, end user connectivity, internet gateways, etc.
 Authentication and authorization: Domain controllers, active directory, Active Directory, Federation Services, SSO integration, etc.
 Compliance requirements and solutions: Standard regulatory and industry-specific compliance Ex: GDPR, HIPAA, PCI DSS, SOX, CCPA, etc. Code scanners, auditing tools, and services.
 Operational models: Development, release & deployment, configuration, escalation, system maintenance, patching, virtualization, etc.
 Environments: DEV, QA, UAT, Pre-Prod, Prod, etc.
 Data Landscape:
· Data Architecture: Lambda, Kappa
· Sources: Web Apps, Mobile Apps, IoT, etc.
· Storage layers: databases, data warehouse, data lakes, delta lakes, file system/share, etc.
· Databases: RDBMS, NoSQL, Analytical, Caching, Big Data management systems, etc.
· ML & AI solutions
· Data Movement/Synchronization: Batch Processing, streaming, pull/push, etc.
· HA-DR and Load Balancing: Agreements (RTO/RPO), tools, services, and native features used for HA, DR, and LB purposes. Ex: Oracle (RAC, Data Guard, Streams, Flashback, Fail-safe, etc.), SQL Server (Always-On, replication, log-shipping, Change Data Capture, etc.), third-party tools (HVR replication, Commvault backups, etc.), Linux/ Windows snapshots, etc.
· Data Integrations and Lineage: Managing and monitoring all data stores, access, classification, and data movement across the enterprise from a central location. It includes data lineage tools (Collibra, metacenter, etc.), data directories, and dictionaries.
· Data Ingestion, ETL, and ELT tools: Kafka, NiFi, Talend, SSIS, Informatica, etc.
· Reporting tools: SSRS, Power BI, Tableau, etc.
· Data Security: Data encryption features, data masking, access control, certificates, key stores/vaults, etc.
· Data Classification:
 Security (Public, Internal, Confidential, Restricted)
 Usage (Hot, Warm, Cool)
 Domain (Government, Health, Financial, PII, etc.)
 Type of Data (Transactional, Operational, Analytical, Master, Historical, system logs, backups, etc.)
 Assessment reports: We need to run assessment tools and services to identify the cloud migration readiness of our workloads.
 Benchmarks: Performance baselines, common errors, and possible resolutions running assessment tools and finding the compatibility issues and migration blockers (Ex: Data Migration Assistant, Upgrade advisor, etc.), etc.
 Software and Hardware Maintenance: Maintenance and upgrade windows, predictive analysis tools, etc.
 Procedures and Policies: Configurations, decommission policies, security frameworks, etc.
 Application rankings: Rank applications and data stores based on their criticality, business dependency, complexity, revenue impact, data volume, etc.
 Business landscape: Operating regions, time zones, runtimes, shifts, critical applications, operational costs, vendors, accountabilities, point of contacts, approval matrix, escalation matrix, domain-specific hardware/software requirements, compliance certifications, etc.
 Documents: AS-IS architecture diagrams, Service Level Agreements, Functional Design Documents, Detailed Design Documents, Commercials, Network and security frameworks, Data Requirements, etc.
· Phase-1 – Output: Documented footprint/inventory of all resources, assets, and processes of the source environment. The discovery document describes the entire enterprise data and infrastructure landscape and migration readiness of the workloads.

Continue reading

Posted in Interview Q&A | Tagged , , , , , , , , | 2 Comments