Your Data Ninjas!
The video on this post shows how we used Collibra's metadata management and governance capabilities to manage a complete ML lifecycle. As metadata piles up in a data science organization around modeling, experimentation and data engineering, providing governance and intelligence capabilities is essential for long term management and oversight.
Adding role based permissions and using Collibra's workflow capabilities for approvals and promotions make it a complete solution.
The California Consumer Privacy Act is a consumer privacy legislation which passed into California law on June 28th of 2018. The bill, also known as “AB 375,” has been described by some as “GDPR of the US.” This act is one of the strongest privacy legislation enacted in any state now, giving more power to consumers in regard to their private data.
It’s just a matter of time before other states will follow suit in the coming years, companies across the U.S. that take proactive steps today to better protect consumer data will be best equipped to ride the waves of change.
These are the three key articles in the law which explains if a business is impacted by CCPA:
An important term loosely defined in the bill is “personal information.” According the AB 375, “The bill…would define ‘personal information’ with reference to a broad list of characteristics and behaviors, personal and commercial, as well as inferences drawn from this information.”
Dozens and perhaps hundreds of specific data items are mentioned in the legislation, including:
There are many steps a business must perform to comply with all facets of the law.
CCPA is the first of probably many steps that companies have to be prepared for with respect to consumer data privacy. It is well worth investing in building processes and automation around finding and categorizing customer data.
Having half-baked solutions or manual solutions will create a lot of churn and manual labor at best and at worst will cause omissions and errors which could cost the company millions of dollars in fines.
It is advisable to work with solution providers who have solved this problem before and have a frameworks and solutions that can be easily reproduced.
Take a look at some of the recent headlines:
“Data Protection Concerns Upend M&A Plans”
“California Passes Sweeping Data-Privacy Bill”
“Marketers Push Agencies to Shoulder More Liability for Data Breaches …”
“Apple CEO Condemns ‘Data-Industrial Complex’”
Truth be told, these headlines are just a fraction of what is happening out in the real world. Data are everywhere (and growing by the minute). All manners of devices have become smarter, or in other words have started producing data.
Networks have multiplied, devices have multiplied, systems have increased. With all this happening, it is important for any organization to build a cohesive, well understood catalog of all information assets.
Data Ninjas recommend a new mantra to live by and that is “you can only protect what you know you have!”. This is an often overlooked by most organizations. This issue is more pervasive in bigger organizations but often is ignored by smaller organizations. While smaller organizations are not exposed to compliance risk, they are exposed to general data privacy and protection risk.
Now let’s look at some of the costs of lack of adequate data protections:
Even if you ignore some of the statistics, the amount of disruption caused by a casual breach to your business would be crippling. Most smaller organizations have threadbare staff on hand for day to day operation so a breach event would be truly catastrophic.
So how can you stop these events from happening in your domain? By being proactive about it. At Data Ninjas, we believe in being prepared (and you don’t have to build all your protections all at once) from the ground up.
Here is a quick plan on how to execute on a data protection/privacy project iteratively:
For organizations making a move to the AWS cloud there are always a lot of questions regarding the storage solution that they should use
As with most services offered by AWS, there are plenty of choices to select from. The selection boils down to a couple of criteria:
The main options to choose from as of 2018 are:
Let’s get into a little more detail on each of them….
If regular flat file or object based storage is what you are looking for, then S3 is the right option to go with. This is a bucket based storage with unlimited capacity where you can store files from 0-5TB of size. Data stored here is secure, durable and highly scalable . S3 uses simple web service interfaces to store and retrieve unlimited amount of data from all over the web. Its built to achieve 99.99% availability and 99.999999999% durability.
Within S3 AWS offers different Storage Tiers
Also called Elastic block storage. These can be thought as disks in the cloud that you can attach to your EC2 instance. These are Storage volumes on the cloud that can be used as a file system or for a DB. They can also in some cases be used as a root volume. There are 5 types of EFS systems:
Snowballs are appliances that are sent to customers who can use them to transfer large amounts of data to the appliance by connecting it locally. The devices are then shipped to AWS where the data is transferred to the AWS infrastructure. It uses multiple layers of security including 256 bit encryption and the appliances are tamper proof.
There are three flavors or Snowballs:
Storage Gateways connects an on premise software appliance with cloud based storage. So your local IT infrastructure is directly connected to AWS storage.
The software appliance can be downloaded as a VM image to be installed at the hosts datacenter. It supports VMWare ESXi or Microsoft Hyper-V.
There are 3 types of Storage gateways
So, as mentioned earlier, there are lots of options to choose from. Based on your needs of size, throughput, accessibility, scale and business requirements you should be able to narrow down one of the options presented above.
Hope this helps you choosing the right AWS cloud storage solution. We can work with you to further help with your decision making process.
Please reach out to us at [email protected] for further enquiries.