joachimhskeie
|
Posted: March 19, 2011 21:50 by joachimhskeie
|
|
Hello, I have proposed a plugin-architecture to BTrace. I would very much like some feedback on the validity and how useful you guys believe such an addition to BTrace would be. Details here: http://kenai.com/jira/browse/BTRACE-64 Very Best Regards, Joachim Haagen Skeie |
BTrace Plugin Architecture
20 posts
Replies: 19 - Last Post: July 30, 2011 10:18
by: joachimhskeie
by: joachimhskeie
showing 1 - 20 of 20
|
joachimhskeie
|
Posted: March 23, 2011 20:54 by joachimhskeie
|
|
Attached my proposal for a plugin architecture to http://kenai.com/jira/browse/BTRACE-64 Updating the attached diagram as well: ![[http://kenai.com/jira/secure/attachment/14034/Btrace+Plugin+Architecture.png]](http://kenai.com/jira/secure/attachment/14034/Btrace+Plugin+Architecture.png "http://kenai.com/jira/secure/attachment/14034/Btrace+Plugin+Architecture.png") I think that such an architecture will greatly improve the extensibility of BTrace! Very Best Regards, Joachim Haagen Skeie |
Jaroslav Bachorik
|
Posted: March 23, 2011 22:04 by Jaroslav Bachorik
|
|
I am sorry I couldn't react earlier. If I read the diagrams correctly the plugin architecture would allow execution of an arbitrary bytecode (in the form of a plugin), effectively bypassing the BTrace strict checks - no loops (to prevent endless loops), no uncontrolled new instances etc. Since there are no restrictions placed on the plugin's instruction set it would run in 'unsafe mode' by default. I don't like this security hole much. I agree that extensibility by plugins would be nice - but I am afraid it will call for more extensive refactorings in the core, vis your hack with introducing BTraceService interface to make the BTraceUtils accessible to plugins due to rather convoluted dependencies in BTrace engine. In fact, I have given some thoughts to the modular BTrace recently - mostly thanks to my experiments with BTrace/JRockit integration ... These are the minimum requirements for a safe modular system I came up with:
As a side effect a proper separation between the BTrace client and server will emerge 
|
|
joachimhskeie
|
Posted: March 23, 2011 22:49 by joachimhskeie
|
|
Hello, I do get the need for strict checks, but on the other hand it would also put the responsibility for ensuring that the plugin offers a safe execution to the plugin developers. Alternatively BTrace could allow the user to specify a "pluginUnsafe=true" option. The way I see it BTrace scripts should be held to a more strict operation-mode than plugins should. If plugins have to adhere to the same strict rules as scripts, then there wont be much need to write a plugin. Plugins would be used to extend the feature set of BTrace, for instance, being able to send off a message to a remote application, touching files on the filesystem, or otherwise performing task that BTrace is unable to provide by itself. I know this sounds dangerous, but I believe that the people developing BTrace plugins would be more concerned with what they are attempting to do, and how they are going to achieve it (safety, errors, etc) than the people developing BTrace scripts. |
|
Jaroslav Bachorik
|
Posted: March 24, 2011 08:10 by Jaroslav Bachorik
|
|
If we leave the plugin code running non-sandboxed wouldn't it be much easier just implement the functionality you want in an external jar (compiling against btrace-core to get access to BTraceUtils), adding the jar on bootclasspath and calling your classes/methods from a BTrace script marked as unsafe? That way an external developer can provide the BTrace script writers with a binary enhancing BTrace functionality and if a writer decides to trust such an extension he can easily use it from an unsafe script. |
|
joachimhskeie
|
Posted: March 24, 2011 08:44 by joachimhskeie
|
|
That might be an option. In my mind, though, a plugin is by definition "unsafe", though. Some restrictions might apply (like not able to start new threads or execute native code, etc.), but any application that has a plugin interface do leave a certain amount of trust to the plugin developers. For instance, I'm sure that its possible to write, i.e. a Firefox plugin that is able to bring down Firefox when executed, resulting that the users will uninstall that plugin for future use. Also, a plugin has to be installed and setup to be included in the BTrace agent parameters (plugindir=/path/to/dir), so a choice on which plugins to trust still has to take place. To summarize it, maybe its possible to offer two plugin modes, one "safe" and one "unsafe" ? I'd rather have the plugins adhere to a standard interface/abstract class from within BTrace, than to mark a script as "unsafe" enabling it to basically call whatever it wants from within BTrace. |
|
Jaroslav Bachorik
|
Posted: March 24, 2011 11:27 by Jaroslav Bachorik
|
|
I can see virtually no difference between putting the dangerous code inside a script or hacking a plugin jar providing an escape route from BTrace strict checks while pretending to adhere to them. We will need to introduce an "almost safe" mode I checked the diffs and the plugins should be loaded by the bootstrap class loader - otherwise they won't be visible from JDK classes eg. Also, it seems that the classpaths for instrumentation are modifiable at any time after the agent has been started. This gave me an idea of specifying the required extension plugins directly in the BTrace script. Something like
@BTrace(
extensionsPath="/path/to/plugins", // optional with a sensible default
requires={"foo.bar.Opener", "bar.foo.Closer"}
)
This would give us the chance to do black magic in BTrace compiler and enable the declared requirements to be accessed directly without failing the compilation process. Also, these arguments will be accessible to the agent so it can use them to adjust class verifier and extend the bootstrap class path (checking for duplicates in requirements and not adding the same jar more than once). The plugin jars would then require additional attributes in their MANIFEST.MF - at least an attribute listing all the extension points it provides to BTrace (eg. BTrace-Provides: foo.bar.Opener, bar.foo.Closer) |
|
joachimhskeie
|
Posted: March 24, 2011 14:50 by joachimhskeie
|
|
I dont see a major difference between the two approaches. In both instances, it is the script developer that chooses to execute code from the plugin, the difference being that in the script-example the classpath is changed during script-execution time, whereas in the plugin example, the classpath is defined at the time the agent loads. In both approaches the BTrace Plugin/Jar developer needs to assure that the code wont do bad things and the script developer need to trust that the plugin/jar wont mess with the runtime of the instrumented application. However, in the plugin implementation it would be easier for the BTrace engine to control the execution, and to handle potential errors, one approach is having the plugin methods throw a potential "BTracePluginException" if anything goes wrong, etc. It would also be possible for the BTrace engine to halt the execution of a plugin if certain curcumstances occur (i.e. deadlock, or a prolonged execution time of the plugin code, etc.). I would agree that BTrace needs to perform more validation on the contents and execution of the plugin, than what is currently implemented, and I like the idea of "almost safe plugin mode"
|
|
Jaroslav Bachorik
|
Posted: March 24, 2011 18:19 by Jaroslav Bachorik
|
|
To be honest, what I am looking for is to bring modularity to the very base of BTrace. My half-a**ed approach at modularizing the monstrous BTraceUtils improved things only very slightly. I would much rather see things like BTraceUtils.Atomic, BTraceUtils.Aggregations etc. as plugins provided in the default BTrace installation. And once doing this I want to keep the boilerplate code at minimum - thus I would prefer not using Plugins.executePluginWithName(name) notation as it would require new plugin for each single method. I am not trying to bash your design - I am sure it would work well for your needs. But the idea of a plugin per functional unit (eg. BTraceUtils.Atomic.getAndGet and BTraceUtils.Atomic.get would require two separate plugin classes) doesn't fit well with the direction I would like the BTrace to take  BTW I like the idea of BTracePluginException Also, the fact that we disassemble and assemble the bytecode of a BTrace script during compilation gives us certain shortcuts to designs otherwise required in Java world. For example you could write Atomic.addAndGet(var) but the BTrace compiler, knowing that the Atomic is a plugin class would generate an equivalent of
try {
Atomic.addAndGet(var)
} catch (BTracePluginException e) {
logException(e)
}
or a more sophisticated wrapper if needed. I don't know whether this approach would help you solve the problems you are trying to solve but if it does I would be really glad if you chose to participate in this modularization approach even if it means some changes to your original design. Also, it would be nice to hear more opinions as we two are rather biased
|
|
joachimhskeie
|
Posted: March 24, 2011 20:10 by joachimhskeie
|
|
I dont mind changing my original design at all. I would be glad to help out with modularizing BTrace. Im going to throw a few ideas out there, hopefully engaging others to join in the discussion as well. Splitting up BTraceUtils into pluggable modules sounds interesting, but also somewhat difficult to do in a general manner I believe. I suppose one approach could be to make BTraceUtils into an abstract (empty)shell - sort of like a marker interface - and have each pluggable component extend BTraceUtils. I guess it would still be possible to use the ServiceLoader to load in the pluggable components. The more difficult piece of the puzzle is to get the BTrace compiler to know which methods each script is able to call. One option would be to make the script declare its intentions in the @BTrace annotation, sort of like
@BTrace(pluginList={"btrace-aggregation", "btrace-string", "btrace-collections"})
The BTrace Compile could then look in its classpath for a jar file matching each feature in the feature list: $agent_home/plugins/btrace-aggregation.jar. I guess btracec and btrace-core could have the standard plugins that are shipped along with BTrace added to the classpath and to the featureList by default, so that only third-party plugins would have to be included in the pluginList like:
@BTrace(pluginList={"eurekaj-proxy"})
eurekaj-proxy.jar would include a META-INF/services/com.sun.btrace.spi.BTracePluginProvider file, as required by ServiceLoader. For this to work, the BTrace engine needs a way to fail safely if, for some reason, a script is compiled against a plugin jar file which is missing during agent runtime - MissingBTracePluginException, log.error, etc. This way, BTrace is very easily extended, the BTrace Engine can enforce that each method in the plugin must declare to throw a BTracePluginException to be made available to the BTrace script, while still letting the BTrace script developer decide which plugins to use in each script. Extending this further I suppose it would be possible for the BTrace core-team to officially approve certain plugins as "safe", via offering certain plugins to become a signed .jar file or similar. That way it would be very easy to distinguish when BTrace is running a plugin in "safe" mode, and when its running a plugin in "unsafe" mode. This whole approach though, makes todays separation between BTraceUtil and BTraceRuntime invalid, as each plugin would need to implement the Runtime-features as well. Some parts of BTraceRuntime should remain though, and that is the initialization of the MXBeans from java.lang.management, and similar static methods. Take all of the above as thoughts. I'm sure you have lot of ideas in this regard as well
|
|
joachimhskeie
|
Posted: March 30, 2011 08:52 by joachimhskeie
|
|
Hello again, Is there a Jira Issue or a description on what you would like to have done in regards to modularization ? |
|
Jaroslav Bachorik
|
Posted: March 31, 2011 08:53 by Jaroslav Bachorik
|
|
Not yet. I think it could even have its own wiki entry linked from the issue. Anyway, I was playing with the verifiers and compiler and created a new branch 'extensions' with very first changes allowing to include a 3rd party code calls in the BTrace scripts. It turns out that in fact it is not necessary to explicitly mention what is an extension class and what not - an annotation placed on the extension class would do as well. if you are interested in trying out the changes just do 'hg update extensions' and it will bring in the correct branch. There is a really simple plugin.test project which builds a jar with a single extension class. In order to use this extension you'll need to define btrace.extension.path pointing to the jar for the target application (eg. you must start the application with -Dbtrace.extension.path=<path_to_extjar>) as well as for the btrace command (modifying the btrace script). Using the system property is just a quick hack to get the things going - it will have to be solved differently. |
|
joachimhskeie
|
Posted: April 01, 2011 07:44 by joachimhskeie
|
I am away over the weekend, but I will have a look at it next week. Regarding the btrace.extension.path could be used to override a default path. BTrace could set up so that all jar files within directory $AGENT_HOME/plugins by default, which could then be overriden with either the -Dbtrace.extension.path, or via the agent parameter -javaagent path/to/agent.jar=extensionPath=/path/to/ext I think it should resolve to a directory instead of a single jar by default, but that the user can select to override it with a single jar if they like.
|
|
joachimhskeie
|
Posted: April 07, 2011 10:07 by joachimhskeie
|
|
Hello, On first view this looks good. Just a quick question. There will be one jar file for each plugin-point ? So BTrace will end up with a btrace-agent-jar, extension/btrace-threads.jar, extensions/btrace-aggregation.jar, etc. ? Or is it possible to list multiple extensions inside META-INF/services for each JAR ? Very Best Regards, Joachim Haagen Skeie |
|
Jaroslav Bachorik
|
Posted: April 07, 2011 14:27 by Jaroslav Bachorik
|
|
It is possible to stuff more actual extensions into one physical extension jar. I've pushed in some recent changes - currently it is possible to create an extension class which can be called from the BTrace scripts (if available from the specified ext locations), register a custom MBeanDecorator (for custom type conversions to OpenType), register a custom CommandReceiver for custom processing of the received commands. Also a new custom command can be introduced and used from plugins. This has allowed me to get rid of the BTraceUtils class and move the functionality to appropriate extensions. The arch wiki page is a bit out of sync as I was busy with the prototype. I will update it ASAP. There are no safety checks yet - but the infrastructure is ready for eg. specifying a set of requested permissions for an extension to be enabled. Since documentation is virtually non existing as of now you should take a look at the plugin.test project - all the new features are used there. Standard extension location is <BTRACE_HOME>/lib/ext - you can provide additional locations containing extension jars using -x switch for btrace command or specifying extPath agent argument. |
|
joachimhskeie
|
Posted: April 20, 2011 19:37 by joachimhskeie
|
| Is there any work remaining for porting existing functionality to extensions, or is the port mainly done ? |
|
Jaroslav Bachorik
|
Posted: April 22, 2011 10:51 by Jaroslav Bachorik
|
|
There are still some things to be done: - verification of the ported functionality / code review - updating samples to use the extensions instead of BTraceUtils - creating a simple migration path from BTraceUtils to extensions for existing scripts It is also important that more people would try using the current version of Extensions API to create new extensions so we can polish the user experience. |
|
joachimhskeie
|
Posted: July 28, 2011 12:02 by joachimhskeie
|
|
Hello again, Holidays are over, and development of the final release of EurekaJ is winding down throughout August, so I have planned to put a bit of work into upgrading my test environment from BTrace 1.2 to BTrace 1.3, and I will have a look at the points suggested above during the process. I also have plans to create an extension, and I will report back on the progress once I have something to report. |
|
joachimhskeie
|
Posted: July 30, 2011 10:18 by joachimhskeie
|
|
Hello, I am trying to make a very simple script using the extensions. The script is as follows:
package org.eurekaj.btracers;
import com.sun.btrace.annotations.*;
import static com.sun.btrace.ext.Collections.*;
import java.util.Deque;
@BTrace
public class CpuInfo {
private static Deque<Long> sysTimeQueue = com.sun.btrace.ext.Collections.BTraceDeque();
private static Deque<Double> totalCpuTimeQueue = com.sun.btrace.ext.Collections.BTraceDeque();
@OnTimer(15000)
public static void printCpu() {
Long timePeriod = com.sun.btrace.ext.Time.millis();
}
}
However, I am not sure how the script is able to access the methods within the extensions. When I try to compile the script above I get: jhs:btraceScripts jhs$ echo $BTRACE_HOME /Users/jhs/Projects/btracemaven/btrace~core-maven/btrace-dist/target/btrace jhs:btraceScripts jhs$ /.//Users/jhs/Projects/btracemaven/btrace\~core-maven/btrace-dist/target/bin/btracec CpuInfo.java >>> located extensions >>> com.sun.btrace.ext.profiling.Profiling >>> com.sun.btrace.ext.Counters >>> com.sun.btrace.ext.Aggregations >>> com.sun.btrace.ext.Reflective >>> com.sun.btrace.ext.Printer >>> com.sun.btrace.ext.sys.Env >>> com.sun.btrace.ext.sys.Memory >>> com.sun.btrace.ext.export.Export >>> com.sun.btrace.ext.Atomic >>> com.sun.btrace.ext.sys.Process >>> com.sun.btrace.ext.Time >>> com.sun.btrace.ext.Strings >>> com.sun.btrace.ext.Collections >>> com.sun.btrace.ext.Numbers >>> com.sun.btrace.ext.sys.VM >>> com.sun.btrace.ext.Threads >>> com.sun.btrace.ext.References CpuInfo.java:4: package com.sun.btrace.ext does not exist CpuInfo.java:4: package com.sun.btrace.ext does not exist CpuInfo.java:9: package com.sun.btrace.ext does not exist CpuInfo.java:10: package com.sun.btrace.ext does not exist CpuInfo.java:14: package com.sun.btrace.ext does not exist error: CpuInfo.java:9:method calls are not allowed - only calls to BTraceUtils are allowed error: CpuInfo.java:10:method calls are not allowed - only calls to BTraceUtils are allowed I have tried different approaches, but it seems com.sun.btrace.ext can be found... Am I doing something wrong here ? |
|
Jaroslav Bachorik
|
Posted: March 24, 2011 18:59 by Jaroslav Bachorik
|
|
For the "almost safe plugin mode" we might implement permissions matching - like in JavaME or Android. A plugin would need to declare the set of permission it needs to run. Using this technique an unknown plugin could be inquired by its potential user for the required permissions and decide whether he goes with the risk or not. |
showing 1 - 20 of 20
Replies: 19 - Last Post: July 30, 2011 10:18
by: joachimhskeie
by: joachimhskeie
