For context: one of the several projects I'm working on right now is an automated extraction system for literate-code-style documentation in python. This isn't the place nor time to talk about the why of it (especially compared to other existing similar solutions). The important thing is the how: it uses a temporary import hook to stub out all module imports, allowing the docs generator to process each module independently at runtime, track imports between them, etc. At the end of the process, it also cleans itself up nicely.
Point being, it's a lot of really complicated fiddling with the python import system. And a lesson I have learned is that messing around with import internals in python is extremely tricky to get right. Furthermore, trying to coordinate correctly between modules that do and don't get modified my the hook is very finicky. Not to mention that supply side attacks on the import system itself could be a terrifying attack vector that would be absurdly difficult to detect.
All this to say, I'm not a big fan of monkeypatching, but I know exactly how it behaves, its edge cases, and what to expect if I do it. It is, after all, pretty standard practice to patch things during python unit tests. And even with all its warts, I would prefer patching to import fiddling any day of the week and twice on Sunday.
Feedback for the author: you need to explain the "why" of your project more thoroughly. I'm sure you had a good reason to strike out in this direction, and maybe this is a super elegant solution. But you've failed to explain to me under what circumstances I might also encounter the same problems with patching that you've encountered, in order to explain to me why the risk of an import hook is justified.
I didn't really get why I'd want to actually use it (vs. just a cool demo) either, until:
> means if you want to make changes to a third-party package, you don't have to take on the maintenance burden of forking, you can package and distribute just your changes.
That's a big win. I've seen and done my share of `# this file from github.com/blah with minor change X to L123` etc.
If the goal is to actually package and distribute the changes via import hook, that makes the supply chain attack question particularly relevant. And it still doesn't explain why you couldn't just package and distribute the monkeypatch itself, instead of creating a whole new import ecosystem surrounding hooks.
I've done my fair share of that too, but I'm still not seeing the benefit vs patching.
Monkey patching an object attribute, such as a method or a function of a module, may affect 3rd party libraries code that use said object.
This solution is interesting, as it provides the patched code as if it were a new package, indendant of the existing one you have installed, like vendoring, but without the burden of it.
In case you want to be the only one seing your patch, this is great. It also makes the whole maintenance easier, as you don't have to wonder if you patch it at the right time or in the right way. MK can fail in many subtle edge cases.
Inheritance, particularly, is a great Mk pitfall I expect this method to transparently work with.
If you only want your own code to see the patch, then why not just wrap it?
I mean if you really need super strong isolation, you can always create a copy of the library object; metaprogramming, dynamic classes, etc, all make it really easy to even, say, create a duplicate class object with references to the original method implementations. Or decorated ones. Or countless other approaches.
My point isn't that I don't see problems that could be solved by this; my point is that I can't think of any problems that this solves, that wouldn't be better solved by things that don't do any innards-fiddling in what is arguably the most sharply-edged part of python: packaging and imports.
And speaking from experience... if you think patching can fail in subtle edge cases, then I've got some bad news for you re: import hooks.
At the end of the day, people who might use this library are looking for a solution to a particular problem. When documenting things, it's really important to be explicit about the pros and cons of your solution, from the perspective of someone with a particular problem, and not from the perspective of someone who's built a particular solution. If I need to drive a nail, and you're selling wrenches, I don't want to hear about all of the features of your wrenches; I want to know if your wrench can drive my nail, and why I would ever want to choose it instead of a hammer.
I can think of a lot of differently-shaped metaphorical nails that fall under the broad umbrella of "I need to change some upstream code but don't want to maintain a fork". And I can think of a whole lot of python-specific specialty hammers that can accomplish that task. But I still can't think of a signle situation where using import hooks to solve the problem is doing anything other than throwing a wrench into a very delicate gearbox. That is the explanation I would need, if I were in the market for such a solution, to evaluate modshim as a potential approach.
The import limitation seems to make this not useful for me. Usually when I am monkeypatching, it's because some code I do not control has a (possibly dynamic) import of the "buggy" module under another name, so I need to make my changes visible under the original name.
If I control all the imports I can usually subclass things myself just fine.
> Because our enhanced Session class now enables retries by default, we don't even need to instantiate it directly. modshim's AST rewriting ensures that internal references within the requests module are updated. This means convenience functions like requests.get() will automatically use our enhanced Session class
This seems to explicitly handle the case you are interested in - automatically updating library-internal references to the lower to instead use the upper?
This is interesting, and I'll try to remember to give this a go next time I'm tempted to patch something from the standard library, but...
The README mentions 3 scenarios that this might be preferred over, but not the fourth which I regularly do: Create my own functions/classes that are composed from the unchanged modules. E.g. a request_with_retries function which adds retry logic to requests without the need to monkey patch. I regularly use decorators as well to add things like retries.
For more complex scenarios Modshim might win out, as mentioned in the understated section of the README "Benefits of this Approach":
> Internal Reference Rewriting: This example demonstrates modshim's most powerful feature. By replacing requests.sessions.Session, we automatically upgraded top-level functions like requests.get() because their internal references to Session are redirected to our new class.
> Preservation of the Original Module: The original requests package is not altered. Code in other parts of an application that imports requests directly will continue to use the original Session object without any retry logic, preventing unintended side-effects.
What I think this means is Modshim lets you really get in to the guts of a module (monkey-patch style, giving you god-like powers), while limiting the damage.
That’s a really cool idea — kind of like OverlayFS but for Python modules. I like how it lets you layer changes on top of existing packages without having to fork or monkey-patch them directly.
The big win here is that it keeps things clean and maintainable — you only ship your changes instead of managing a full fork, and you don’t mess up the global namespace. It also makes experimenting with tweaks a lot easier.
The tricky parts might be keeping import behavior consistent and making sure debugging still works nicely since AST rewriting can sometimes make stack traces a bit messy.
Overall, it’s a clever middle ground between monkey-patching and forking — really nice concept.
This feels too much like breaking the guarantee sticker of a vendor code, and if your vendor pushes updates weekly, or daily, you are stuck pushing updates to your shimmed code, which officially becomes "unnamed fork". Even for tests, let's say that they changed an input type, I don't see an improvement in my workflow: I still need to update my "unnamed fork". At least with a fork I get to see the whole git history, including my contributions, and testing with monkey patching helps me create clear setUp and tearDown steps.
When you have a scalpel, you give it to operating doctors during the operation, not to 5 year olds on the street.
Yeah, but the example of "*.retries(...)", in the context of "import some_login_library.Login(...)" is quite powerful! It basically looks like a "super-decorator", and I can definitely see the utility of effectively re-compiling a (third-party) module at runtime to handle something that's more unique to your use case.
Your patch "with retries" might never be accepted, and maintaining any kind of fork(s) or "out-of-tree patches" is not as integrated into the programming environment. Being able to say "assert WrappedLoginLibrary().login(), '...with retries...'" keeps you testable and "in" the language proper.
Monkey-patching is altering the code at runtime, not the source code, so from the "alternative to forking and modifying" part it doesn't sound like an alternative to that.
Edit: okay Readme is clear on it and the description does make sense, the short description here just confused me.
> * Fix bugs in third-party libraries without forking
> * Modify the behavior of existing functions
> * Add new features or options to existing classes
> * Test alternative implementations in an isolated way
only the last sounds close to something you might actually want to do, and then only as a throwaway thing
If you want to change a library, fork it. If you want to change the behavior of existing functions, don't or at least fork first. If you want to add new features to a class, write a new class, or again, at least fork first
Point being, it's a lot of really complicated fiddling with the python import system. And a lesson I have learned is that messing around with import internals in python is extremely tricky to get right. Furthermore, trying to coordinate correctly between modules that do and don't get modified my the hook is very finicky. Not to mention that supply side attacks on the import system itself could be a terrifying attack vector that would be absurdly difficult to detect.
All this to say, I'm not a big fan of monkeypatching, but I know exactly how it behaves, its edge cases, and what to expect if I do it. It is, after all, pretty standard practice to patch things during python unit tests. And even with all its warts, I would prefer patching to import fiddling any day of the week and twice on Sunday.
Feedback for the author: you need to explain the "why" of your project more thoroughly. I'm sure you had a good reason to strike out in this direction, and maybe this is a super elegant solution. But you've failed to explain to me under what circumstances I might also encounter the same problems with patching that you've encountered, in order to explain to me why the risk of an import hook is justified.
> means if you want to make changes to a third-party package, you don't have to take on the maintenance burden of forking, you can package and distribute just your changes.
That's a big win. I've seen and done my share of `# this file from github.com/blah with minor change X to L123` etc.
I've done my fair share of that too, but I'm still not seeing the benefit vs patching.
This solution is interesting, as it provides the patched code as if it were a new package, indendant of the existing one you have installed, like vendoring, but without the burden of it.
In case you want to be the only one seing your patch, this is great. It also makes the whole maintenance easier, as you don't have to wonder if you patch it at the right time or in the right way. MK can fail in many subtle edge cases.
Inheritance, particularly, is a great Mk pitfall I expect this method to transparently work with.
I mean if you really need super strong isolation, you can always create a copy of the library object; metaprogramming, dynamic classes, etc, all make it really easy to even, say, create a duplicate class object with references to the original method implementations. Or decorated ones. Or countless other approaches.
My point isn't that I don't see problems that could be solved by this; my point is that I can't think of any problems that this solves, that wouldn't be better solved by things that don't do any innards-fiddling in what is arguably the most sharply-edged part of python: packaging and imports.
And speaking from experience... if you think patching can fail in subtle edge cases, then I've got some bad news for you re: import hooks.
At the end of the day, people who might use this library are looking for a solution to a particular problem. When documenting things, it's really important to be explicit about the pros and cons of your solution, from the perspective of someone with a particular problem, and not from the perspective of someone who's built a particular solution. If I need to drive a nail, and you're selling wrenches, I don't want to hear about all of the features of your wrenches; I want to know if your wrench can drive my nail, and why I would ever want to choose it instead of a hammer.
I can think of a lot of differently-shaped metaphorical nails that fall under the broad umbrella of "I need to change some upstream code but don't want to maintain a fork". And I can think of a whole lot of python-specific specialty hammers that can accomplish that task. But I still can't think of a signle situation where using import hooks to solve the problem is doing anything other than throwing a wrench into a very delicate gearbox. That is the explanation I would need, if I were in the market for such a solution, to evaluate modshim as a potential approach.
If I control all the imports I can usually subclass things myself just fine.
This seems to explicitly handle the case you are interested in - automatically updating library-internal references to the lower to instead use the upper?
If A is my application, B is buggy, and C is some other library, consider:
The README mentions 3 scenarios that this might be preferred over, but not the fourth which I regularly do: Create my own functions/classes that are composed from the unchanged modules. E.g. a request_with_retries function which adds retry logic to requests without the need to monkey patch. I regularly use decorators as well to add things like retries.
For more complex scenarios Modshim might win out, as mentioned in the understated section of the README "Benefits of this Approach":
> Internal Reference Rewriting: This example demonstrates modshim's most powerful feature. By replacing requests.sessions.Session, we automatically upgraded top-level functions like requests.get() because their internal references to Session are redirected to our new class.
> Preservation of the Original Module: The original requests package is not altered. Code in other parts of an application that imports requests directly will continue to use the original Session object without any retry logic, preventing unintended side-effects.
What I think this means is Modshim lets you really get in to the guts of a module (monkey-patch style, giving you god-like powers), while limiting the damage.
https://github.com/epgui/pybond
The big win here is that it keeps things clean and maintainable — you only ship your changes instead of managing a full fork, and you don’t mess up the global namespace. It also makes experimenting with tweaks a lot easier.
The tricky parts might be keeping import behavior consistent and making sure debugging still works nicely since AST rewriting can sometimes make stack traces a bit messy.
Overall, it’s a clever middle ground between monkey-patching and forking — really nice concept.
When you have a scalpel, you give it to operating doctors during the operation, not to 5 year olds on the street.
Your patch "with retries" might never be accepted, and maintaining any kind of fork(s) or "out-of-tree patches" is not as integrated into the programming environment. Being able to say "assert WrappedLoginLibrary().login(), '...with retries...'" keeps you testable and "in" the language proper.
Edit: okay Readme is clear on it and the description does make sense, the short description here just confused me.
> * Fix bugs in third-party libraries without forking
> * Modify the behavior of existing functions
> * Add new features or options to existing classes
> * Test alternative implementations in an isolated way
only the last sounds close to something you might actually want to do, and then only as a throwaway thing
If you want to change a library, fork it. If you want to change the behavior of existing functions, don't or at least fork first. If you want to add new features to a class, write a new class, or again, at least fork first