A discussion between a CxO and a senior Data Architect Part 2

A discussion between a CxO and a senior Data Architect Part 2

Links to other parts

A discussion between a CxO and a senior Data Architect Part 1

A discussion between a CxO and a senior Data Architect Part 3

A discussion between a CxO and a senior Data Architect Part 4

A discussion between a CxO and a senior Data Architect Part 5

.

Background: We have been going through a discussion that took place between senior leadership and a data architect. They took a coffee break and here Part – 2 continues.

.

<During the break, had some casual talks about personal life and hobbies, etc. which is irrelevant here. We’ll see the discussion after the break>

.

Discussion Follows:

.

Alison: So, Vasumat, let’s say a CxO level employee asked you “why we should migrate to the Cloud?”. Assume that person doesn’t care about technology, but business. You need to convince him/her. Can you?

Vasumat: Well, that’s a reasonable question and we call it a Business Case. That’s what we do in the 0th phase “Migration Preparation and Business Planning”. Before starting the migration project, we need to establish a Business Case for migration, and set expectations for cost, return, and the timing of those costs and returns. Money is everything in business, so the migration business case showcases the TCO (Total Cost of Ownership) and ROI (Return On Investment) calculations that tell the benefit of migrating to the cloud.

Alison: Can you define the terms TCO and ROI?

Vasumat: Sure! Those are the key parameters to establish the business case. TCO and ROI both talk about money we need to pay from our pocket and the gain or return that we get on the investment.

Alison: I am listening….

Vasumat:

 TCO (Total Cost Ownership): is the sum of the expenses associated with purchasing, deploying, using, and retiring a product/equipment/asset. Essentially, the sum of the purchase and operating price of an asset for its lifetime. There are two types of costs involved direct and indirect. Ex: When we buy a car:
 Direct cost: Car retail price, registration tax, road tax, insurance premium, excise duty, state tax, monthly fuel expenses, car maintenance and servicing cost, etc.
 Indirect Cost: Parking tickets and toll charges come under hidden or indirect charges.
 Car TCO: The total of all Direct and Indirect expenses is called the car TCO. Likewise, we need to calculate the overall TCO for our organization’s technology infrastructure which includes both hardware, software, and operations costs.
 On-premises TCO calculation: We usually perform infrastructure audits to identify the direct and indirect costs
 Direct costs:  Direct costs are relatively easy to calculate, as they remain on our organization’s balance sheet. It is categorized into hardware (physical assets), software (licensing, maintenance, upgrades, etc.), management (architect, hosting, reporting, etc.), support (supporting staff), and implementation (development, customizing, and integrating). Ex: Physical servers, storage devices, network devices, printers, software licenses, maintenance contracts, warranties, supplies, material, spare parts, real estate/space, internet facility, labor/resources, hiring, onboarding, training employees, maintenance, upgrades, security, disaster recovery, etc.
 Indirect costs: Difficult to calculate but equally important as direct costs. Ex: Un-scheduled maintenance due to breakdown or failure, business impact due to unexpected downtime, power, and energy consumptions, depreciation of asset performance and cost over time, etc.
 Cloud TCO calculation: Almost all the cloud providers are offering an advanced TCO calculator. Based on our on-premises analysis we can input the infrastructure details (servers, OS, cores, memory, storage, licensing, etc.) and it will automatically calculate and report the estimated cloud TCO. Additionally, we need to consider the one-time investment in migration activity.
 Direct: Cloud services & subscription, integration & testing, consulting & training, and Migration cost. Ex: Database service, App service, VM, Load balancer, data ingestion rates, storage, application testing, training on cloud technology, hiring for cloud migrations, the cost for 6R strategy (ex: upgrading to new versions), etc.
 Indirect: Excess compute and storage reservations, premium service packages with no proper SLA (no guarantee on gains Ex: performance, cost, support), software licenses used or forfeited (loosing or giving up) in the transition, required infrastructure at on-premises after the cloud migration (hybrid model), hosting without POC can cause unexpected cost increments, and user behavior (untrained users may provision a lot of unnecessary services without knowing the consequences from cost point), etc.
 ROI (Return on investment): It tells us how much profit or loss our investment has earned. ROI is the ratio of a profit or loss made in a fiscal year, expressed as a percentage. Here is the simple formula:
 General formula: ROI = (Net Profit / Cost of Investment) x 100
 *Net Profit = Gain from the investment – Cost of Investment

.

 ROI formula commonly used in cloud migrations:
 ROI = ((FVI – IVI) / Cost of Investment) X 100
 *FVI: Final Value of Investment
 *IVI: Initial Value of Investment

Continue reading

Posted in Interview Q&A | Tagged , , , , , , , , , , , | Leave a comment

A discussion between a CxO and a senior Data Architect Part 1

.

A discussion between a CxO and a senior Data Architect Part 1

Links to other parts

A discussion between a CxO and a senior Data Architect Part 2

A discussion between a CxO and a senior Data Architect Part 3

A discussion between a CxO and a senior Data Architect Part 4

A discussion between a CxO and a senior Data Architect Part 5

.

.

Below is the conversation that happened in an interview for the role of Lead Data Architect. It was more like business analysis and technical discussion between a client and a consultant. We try our best to narrate the same way as it happened in the interview. We are changing the names for privacy purposes, assuming that the interviewer is Alison, and the interviewee is Vasumat.

Mr. Vasumat has already gone through 1 coding round and 3 technical discussions within the span of 8 weeks. Now, he is having the final discussion with senior leadership. He was informed that he may need to prepare to spend 6 to 8 hours (including lunchtime) in their office. Vasumat reached the office and completed his entry formalities by 8:45 AM whereas his interview is scheduled for 9:15 AM. HR introduced Vasumat to the interviewer.

.

Here you go:

.

Alison: It’s nice to meet you Mr. Vasumat

Vasumat: Honor is all mine mam

Alison: You can call me Alison.

 I am a chief program director working with XXXXX for the last 20 years and I take care of the digital transformation and modernization of our software applications.
 We have a humongous enterprise that includes thousands of servers, databases, and petabytes of data with heterogeneous database systems. We’ve been planning to migrate our applications to the cloud.
 That’s the reason I’ve been staying here in India for the next few months to settle down the tech stack and for the migration kickoff. We hired cloud architects and looking for a data architect who can handle end-to-end data migration.
 We have a partnership with Microsoft, so Azure will be the primary choice. However, down the line, we would like to go with multi-cloud architecture most probably with Azure and Google.
 Richard told me that you have considerable experience, especially in data platforms and cloud migrations. Also, from the previous discussions, they have a positive outlook on your profile. I just wanted to discuss it before finalizing.
 So Vasumat, I would like to hear about your experience with cloud migrations

.

Vasumat: Thanks for the detailed background. I have over 13 years of experience in dealing with enterprise database systems. My cloud journey started 7 years back and earned extensive knowledge in migrating heterogeneous database systems from on-premises to the cloud. Experience in both Azure and AWS but my primary specialization is in Azure.

Alison: Since you are an architect, you must have participated in the migration planning and execution, isn’t it?

Vasumat: Certainly! It’s part of my role.

.

Alison: Great, can you describe the phases of cloud migration at a high level?

Vasumat: There are 5 (1+4) phases involved:

 Migration Preparation and Business Planning + Discovery, Planning, Execution, and Optimization.

Alison: I am listening…

Vasumat: Let me explain in detail.

 Migration Preparation and Business Planning: We create a business case for migration by projecting benefits in terms of cost, performance, and digitalization aspects and seek business approvals. Before starting the next phase, we must have a dedicated stakeholder (Cloud/Solution/Migration/Data Architect) who can lead and execute the migration project.
 Phase 0 – Output: A clear Business Case that tells us why we are migrating to the cloud

.

 Discovery & Assessment: We understand, analyze, and collect data from the source environment. This includes infrastructure, software, hardware, network & security, operational models, enterprise policies and agreements, and data landscape. I do take care of the data part.
 Software applications: web, desktop, mobile, message broker, web service, etc.
 Application dependencies: Custom libraries, parent and child apps, integrations, etc. Ex: Our web app is using a Currency conversion service; we built a custom library to convert an invoice query result to a PDF etc.
 Legacy applications: Unsupported / Older versions, mainframes, etc.
 Third-party solutions: Collibra, Meta-Center, HVR replication, Commvault, etc.
 Client Tools: Visual Studio, Eclipse, Oracle developer, SSMS, MySQL Workbench, etc.
 Source code repository: TFS, SVN, Git, Bitbucket, etc.
 Operating systems and capacity: Windows Linux, Mac, Memory, CPU, IO, Disks, etc.
 Licensing: Opensource, corporate agreements, renewal status, licensing mobility, etc.
 Hardware mapping: Servers, Racks, Cables, Switches, Storage, Routers, Power Equipment, Colling Systems, Network devices, routers, etc.
 Servers: Physical / Virtual, Application, Web, Database, Proxy, Mail, FTP, Print, DNS, etc.
 Network and Security: SSL, TLS, Encryption, firewalls, network security groups, web application and third-party firewalls, end user connectivity, internet gateways, etc.
 Authentication and authorization: Domain controllers, active directory, Active Directory, Federation Services, SSO integration, etc.
 Compliance requirements and solutions: Standard regulatory and industry-specific compliance Ex: GDPR, HIPAA, PCI DSS, SOX, CCPA, etc. Code scanners, auditing tools, and services.
 Operational models: Development, release & deployment, configuration, escalation, system maintenance, patching, virtualization, etc.
 Environments: DEV, QA, UAT, Pre-Prod, Prod, etc.
 Data Landscape:
· Data Architecture: Lambda, Kappa
· Sources: Web Apps, Mobile Apps, IoT, etc.
· Storage layers: databases, data warehouse, data lakes, delta lakes, file system/share, etc.
· Databases: RDBMS, NoSQL, Analytical, Caching, Big Data management systems, etc.
· ML & AI solutions
· Data Movement/Synchronization: Batch Processing, streaming, pull/push, etc.
· HA-DR and Load Balancing: Agreements (RTO/RPO), tools, services, and native features used for HA, DR, and LB purposes. Ex: Oracle (RAC, Data Guard, Streams, Flashback, Fail-safe, etc.), SQL Server (Always-On, replication, log-shipping, Change Data Capture, etc.), third-party tools (HVR replication, Commvault backups, etc.), Linux/ Windows snapshots, etc.
· Data Integrations and Lineage: Managing and monitoring all data stores, access, classification, and data movement across the enterprise from a central location. It includes data lineage tools (Collibra, metacenter, etc.), data directories, and dictionaries.
· Data Ingestion, ETL, and ELT tools: Kafka, NiFi, Talend, SSIS, Informatica, etc.
· Reporting tools: SSRS, Power BI, Tableau, etc.
· Data Security: Data encryption features, data masking, access control, certificates, key stores/vaults, etc.
· Data Classification:
 Security (Public, Internal, Confidential, Restricted)
 Usage (Hot, Warm, Cool)
 Domain (Government, Health, Financial, PII, etc.)
 Type of Data (Transactional, Operational, Analytical, Master, Historical, system logs, backups, etc.)
 Assessment reports: We need to run assessment tools and services to identify the cloud migration readiness of our workloads.
 Benchmarks: Performance baselines, common errors, and possible resolutions running assessment tools and finding the compatibility issues and migration blockers (Ex: Data Migration Assistant, Upgrade advisor, etc.), etc.
 Software and Hardware Maintenance: Maintenance and upgrade windows, predictive analysis tools, etc.
 Procedures and Policies: Configurations, decommission policies, security frameworks, etc.
 Application rankings: Rank applications and data stores based on their criticality, business dependency, complexity, revenue impact, data volume, etc.
 Business landscape: Operating regions, time zones, runtimes, shifts, critical applications, operational costs, vendors, accountabilities, point of contacts, approval matrix, escalation matrix, domain-specific hardware/software requirements, compliance certifications, etc.
 Documents: AS-IS architecture diagrams, Service Level Agreements, Functional Design Documents, Detailed Design Documents, Commercials, Network and security frameworks, Data Requirements, etc.
· Phase-1 – Output: Documented footprint/inventory of all resources, assets, and processes of the source environment. The discovery document describes the entire enterprise data and infrastructure landscape and migration readiness of the workloads.

Continue reading

Posted in Interview Q&A | Tagged , , , , , , , , | 2 Comments

Create a Virtual Machine on your laptop for free

Create a Virtual Machine on your laptop for free

The post “Create a Virtual Machine on your laptop for free” takes you through a step by step process of creating a virtual machine on your laptop while not touching the existing host operating system. In recent times I’ve got plenty of questions from various blog followers on how to set up a VM with less or no cost. Microsoft lovers merely tend to deal with Windows OS and we never try to explore options around Linux. But sometimes we may need to perform POC’s that require opensource support. Now, we’ll see how to set up a VM on your local machine with Ubuntu 18.4 Operating System.

There are multiple ways to set up a VM:

  1. Use AWS / Azure one-year free tier
  2. Host a shared Linux VM on a datacenter just with the price of a caramel chocolate
  3. Use Oracle Virtual Box and create a virtual machine on your laptop for free
  4. Other options: VMWare Workstation, Windows 10 Hyper-V

Considering the limitations and feasibility, I vote for the third option. We are going to create our own VM on a local machine using the product Oracle Virtual Box.

Recommended resources for VM:

  1. Host OS (OS on your laptop/system): Windows 10 Operating System
  2. Available RAM: 1/2 GB
  3. Available Space: 5/10 GB
  4. Guest OS: CentOS / Ubuntu ISO image
  5. Administrator access on the Host system

Oracle Virtual Box:

Virtual Box is a virtualization product that absolutely supports both an individual and an enterprise user. Few things I really liked are “it’s an opensource”, “Optimized”, “Lightweight” and “OS support”. If you have a proper host machine (Laptop/Desktop), you can configure multiple VM’s with different OS and it is very handy to perform the POC’s for your business requirements.

Note: VM supports various OS (Ex: Linux, Windows, Mac, etc). For a quick POC I am going to create a VM with Ubuntu Desktop OS.

Now, we are going to start creating a VM on my local machine:

  • Install Oracle Virtual box
  • Download Ubuntu 18.4 OS software
  • Create a new VM the OS Ubuntu

Install Oracle Virtual Box:

It is the easiest task in the given list. Just download the Oracle Virtual Box from here. Since it is an opensource, it is licensed under the General Public License. From the given link, just click on the image “Download VirtualBox 6.1” and it redirects you to next page from where you can download the installation package by clicking on “Windows Hosts”. It downloads a VirtualBox-6.1.12-139181-Win.exe file sized 103 MB.

Create a Virtual Machine on your laptop for free Create a Virtual Machine on your laptop for free

I am going to install Virtual Box from downloaded “.exe” file and I would like to take the defaults:

Continue reading

Posted in Miscellaneous | Tagged , , , , | 6 Comments