This video gives insight on how a platform can be built on databricks . This is essentially a library that automates the table creation , airflow code generation etc .
this next video is about sputnik which is opensource courtesy – airbnb – it also provides ways for job logic and run logic
One of the challenges many face in using compute available from Cloud providers to enable machine learning , is that the training Data has to be uploaded to the cloud. A lot of organizations are not comfortable uploading sensitive data to the cloud .
Homomorphic encryption can help overcome this challenge . This encryption allows computation to be performed on encrypted data . The final result can be decrypted with the private key and it will return the same result as if the model was built with unencrypted data. This open up the potential for the organization to encrypt the training data on prem. The encrypted data can then be uploaded to the cloud and machine learning model can be trained and built in the cloud. The model can then predict and output the result in encrypted format which can then be decrypted on prem with the private key. This ensures that only encrypted data is pushed to the cloud thus significantly reducing the risk. This allows organization to leverage the vast computing power that’s available in the cloud.
Here is a simple example of homomorphic encryption . first step is to install the phe package
pip install phe
The next step is to write a simple python program to demonstrate the addition of two numbers
import phe as paillier
print("generating paillier keypair")
pubkey , prikey = paillier.generate_paillier_keypair(n_length=64)
a = pubkey.encrypt(10)
b = pubkey.encrypt(20)
c = a + b # adding two encypted values - these are two objects
print ( "adding the encrypted values , the output would be another encrypted object")
print(c)
print(" decrypt with private key")
print(prikey.decrypt(c))
the output of this program is as follows
generating paillier keypair
adding the encrypted values , the outout would be another encrypted object
<phe.paillier.EncryptedNumber object at 0x0000016D8252DCD0>
decrypt with private key
30
the output of adding 10 and 20 is 30 , even though the sum was done on encrypted objects
This is a very simplistic example of homomorphic encryption . The next step is to use this to build an actual model on encrypted data .