It's dangerous that we inject connection code into pyspark notebooks for connecting to messagehub. Does spark streaming even support messagehub with python? This post suggests it doesn't: https://issues.apache.org/jira/browse/SPARK-16534. The danger here is that we are going to lead users into spending a lot of time trying to connect to message hub from pyspark when that may never work. A PR was provided to implement the functionality but was rejected because of pyspark streaming issues: https://github.com/apache/spark/pull/14340
Why is it useful?
|Who would benefit from this IDEA?|
How should it work?