| Type | Issue | Status | submitted | Date | 8-Mar-2014 07:15 |
|---|---|---|---|---|---|
| Version | r3 master | Category | Unspecified | Submitted by | BrianH |
| Platform | All | Severity | minor | Priority | normal |
| Summary | [Epic] Finalize the core semantics of the module system |
|---|---|
| Description |
Rebol's module system was originally meant for three main reasons: - To enable the programming-in-the-small that regular developers are used to. - To enable the kind of programming-in-the-large that advanced developers need. - To help organize the implementation of Rebol itself. So Carl made a module system, based on his initial proposal. And then I filled in the gaps, for the initial working version in 2.100.80. The code is largely based on my module system that I made in 2000, with some syntax inspiration from Gabriele's modules used by Qtask and PowerMezz. Then, for 2.100.108, Carl came up with a better model for script/module options and settings, and requested new features like script compression, checksums, and length-specified embedding for multi-scripts. Which I implemented by doing a full rewrite of the whole thing (Carl set the code style and wrote part of sys/load-header and sys/load-ext-module, and tested everything with his test suite). And that is what we've been using since, a work in progress which had a native bug blocking its full use until Rebol was open sourced. As of Rebol 2.100.108 (barring bugs), we had programming-in-the-small covered with the script model. However, those bugs and other issues blocked the other two. Now, finally, we are able to do what is necessary to enable the rest, and have figured out how to do it. The core model is strong, we just need to switch the defaults. We need to change which kind of module is the "regular" module. We have some developers in the community who are familiar with the needs of programming-in-the-large, and who even have experience with doing it in Rebol, despite Rebol 2 having not been inherently suitable for that kind of thing. After discussions with several of these, we've determined that those developers need to have strong modularity by default. The module system has supported strong modularity since 2.100.80, but the default has been the modules that manage the runtime library lib. We need to change that. Open sourcing Rebol has changed what we need from internal modularity as well, because it has changed the needs of the user base, because the number of new users is greater than the old. It is now more important than ever to make a clean start. And that means, one of the most important driving forces of internal modularity is being able to selectively enable the backwards-compatible stuff that we really need to change going forward, so that we can actually make those changes. That means that in many cases, the need to selectively override lib is as important as the need to manage lib. Andreas Bolka, Maxim Olivier-Adlhoch and I have discussed the changes needed. Andreas and I recently determined that they were very minimal. Semantically, they're practically just tweaks. But with these changes, Rebol will be ready for the needs of real modular developers, while not breaking the model needed for scripts. This is the list of those changes. Related tickets in the first comment, to make linking easier. Status will be updated there. |
| Example code |
| Assigned to | n/a | Fixed in | - | Last Update | 8-Mar-2014 11:18 |
|---|
| Comments | |
|---|---|
|
(0004354)
BrianH 8-Mar-2014 07:15 |
New model:
- Change the precedence of import: #1998 - Change the default module type: #2115 - Selective export, same import: #2114 - Manage your private imports: #2116 - Clean up the terminology: #1807 - Change the story: New conceptual model for the same semantic distinctions. Outline in a comment below. Wording of the above tickets might need adjustment. Multi-scripts: - Explain them: What they are and why they're needed (hint: extensions, encapsulation, interoperability). Outline in a comment below. - Fix them: #1877 and #1941. Tickets may need rewriting. - Make them faster: #1915 and #1916. - Design an API for them: Needs experimentation, maybe competing tickets. Related fixes: - Make SUFFIX? url more useful #2136 - it's used by LOAD for file type detection. Test all of the things: - Module tests and examples in rebol/rebol, along with the rest of the test suite. What the new approach lets us do: - Internal modules, even for natives: #2126 - How to get our old stuff back: The 'old module #2132. Don't complain, it's still there if you need it. - Better extensions, with progressive APIs. #1998 merged. #1877, #1941 and #2136 in PRs. #1807, #2114, #2115, #2116 fully designed. #2126 verified possible. |
|
(0004355)
BrianH 8-Mar-2014 08:07 |
The New Default
Real programming-in-the-large needs strong modularity. We have strong modularity. What we need is to make that style the default, "normal". That means that the old distinction between "regular" and "private" modules needs to change, as does our whole way of thinking about what each style is for, and how we even talk about them. Private is the new normal. Normal modules are used for modular programming. They follow and encourage the principles of strong modularity, and have the benefits of that: localized, selective import; local overriding; multiple loaded versions; mixin semantics. For internal organization, optional or utility APIs, varying compatibility needs in the same code base, these are the way to go. However, if you want to use normal modules, you need to declare what you need, since they export directly into a place that is local to where they're requested from. That's the strong-modular way. REBOL [type: module needs: [fun-stuff]] export my-awesome-function: func [] [print "Hello world!"] Similar ideas: "modules" or "units" from Modula, Ruby, Delphi, and many other languages. However, for programming-in-the-really-large, you need to manage the entire application. An application can be made up of large, separately developed collections of code, written by different groups of developers. Internally these may be organized in very different ways, and even require different versions of the same support modules. This is component-oriented programming. Similar ideas: Java's JAR and WAR files, .NET assemblies, Ruby's gems. Components need to be collected and conflicts resolved on an application level. They frequently manage internal state, so multiple running versions of the same component would be a bad idea - you may want multiple APIs to the same state though. Sometimes, they need to be upgradable, which means being able to migrate that state to the new version. To do all that you need a central place where all of these overrides and updates can be managed: you need a runtime library. Our runtime library is called "lib", and its main feature is that there is only one lib, and everyone imports from lib. That makes it possible to manage on a global basis, override things, update things. With lib, you can have the control your application needs to combine components, upgrade them, and all sorts of tricks. We need a kind of module for managing lib, which we call a "lib module". Lib modules are special, for the simple reason that they export to lib instead of some private destination. Their whole reason for existence is to add stuff to the runtime library. For simple lib modules this may be no different from normal modules, but advanced lib modules can do advanced tricks like overriding the exports of older versions of themselves, or migrating the old data to the new module, to be served up by a new API or new internal model for a compatible API. The simple reason they can do all this stuff is because they can get access to the previous exports and change them - you can't necessarily get access to stuff that was exported privately. You usually specify the lib modules your application needs in one place, so you can manage overrides and deal with conflicts just once. This is similar to a Gemfile in Ruby, and many other languages have similar concepts. Once your lib modules are loaded you never need to refer to them in the Needs headers or IMPORT statements anywhere else in your app. Once the exports are in lib, they're there. You only need to explicitly import regular modules when you need them; lib modules become part of the runtime. You can also have your lib module be the external interface of a component that is internally made up of a bunch of private modules. They can even all be in one file, as MODULE expressions, or the result of a module-aware preprocessor that converts multiple modules into one package. The module syntax was specifically designed for this. If you think this is unnecessary, remember that extensions contain just such a packaged collection of modules. One gotcha with lib modules: They need a name. In order to know whether they are loaded, whether they have done the things to lib that they need to, you have to have some way of referring to the module, and that's what a name is: a way to refer to something. Without that bit of information it's just not safe to let things export to lib, because you won't have a way to easily check whether it's happened yet. So they need a name. REBOL [name: awesome type: module needs: [fun-stuff] options: [lib]] export my-awesome-function: func [] [print "Hello world!"] With a name, you don't have to load modules multiple times, you can just get to the module that's already loaded. Because of that benefit, normal modules also usually have names, even though they don't necessarily require them. For that matter, if a named module doesn't export anything then there is no difference between a normal or lib module at all. The biggest win we get from the lib model is that it allows us to support that old standby of programming-in-the-small: scripts. Scripts don't have to care where their stuff comes from - regular modules, demand or preloaded lib modules, internal functions, it's all the same to scripts. Heck, scripts don't even need headers. You can even enter scripts interactively in the console, every command being a different script. It's funny that in a strongly modular, even component-based system, the most advanced trick of all is seamlessly supporting old-school seemingly non-modular scripts for programming in the small. |
|
(0004356)
BrianH 8-Mar-2014 08:07 |
Placeholder for multi-scripts outline. |
| Date | User | Field | Action | Change |
|---|---|---|---|---|
| 2-Mar-2015 11:34 | BrianH | Comment : 0004354 | Modified | - |
| 2-Mar-2015 11:26 | BrianH | Comment : 0004354 | Modified | - |
| 2-Mar-2015 10:13 | BrianH | Comment : 0004354 | Modified | - |
| 15-Mar-2014 17:52 | BrianH | Comment : 0004354 | Modified | - |
| 8-Mar-2014 11:18 | BrianH | Description | Modified | - |
| 8-Mar-2014 11:17 | BrianH | Description | Modified | - |
| 8-Mar-2014 11:15 | BrianH | Description | Modified | - |
| 8-Mar-2014 11:10 | BrianH | Description | Modified | - |
| 8-Mar-2014 10:50 | BrianH | Comment : 0004355 | Modified | - |
| 8-Mar-2014 10:46 | BrianH | Comment : 0004354 | Modified | - |
| 8-Mar-2014 09:50 | BrianH | Comment : 0004355 | Modified | - |
| 8-Mar-2014 09:49 | BrianH | Comment : 0004355 | Modified | - |
| 8-Mar-2014 08:07 | BrianH | Comment : 0004356 | Added | - |
| 8-Mar-2014 08:07 | BrianH | Comment : 0004355 | Added | - |
| 8-Mar-2014 08:06 | BrianH | Comment : 0004354 | Modified | - |
| 8-Mar-2014 07:15 | BrianH | Comment : 0004354 | Added | - |
| 8-Mar-2014 07:15 | BrianH | Ticket | Added | - |