How do I write into object storage from spark program
Hi,
I tried the following code and it gives me an error "Invalid Response: Method POST on https://identity.open.softlayer.com/v2.0 failed, status code: 500, status line: HTTP/1.1 500 Internal Server Error"
I also tried different "...auth.url" but none of them worked.
Configuration conf = sc.hadoopConfiguration();
String prefix = "fs.swift.service." + SERVICE_NAME;
conf.set(prefix + ".auth.url", "https://identity.open.softlayer.com/v2.0");
conf.set(prefix + ".auth.endpoint.prefix", "endpoints");
conf.set(prefix + ".tenant", "");
conf.set(prefix + ".username", "");
conf.set(prefix + ".password", "");
conf.set(prefix + ".region", "dallas");
conf.setBoolean(prefix + ".public", true);
conf.setInt(prefix + ".http.port", 8080);
DataFrame df = //...
df.write().parquet("swift://notebook." + SERVICE_NAME + "/" + CONTAINER_NAME + "/test.parquet");