[PonyORM-list] Pony's Cache Size Limit?

[Gamesbrainiac]

On Sun, Nov 9, 2014 at 6:08 PM, Matthew Bell <matthewrobertbell at gmail.com>
wrote:

> You were right, my code was lazily thought out. With one query (with one
> subquery), the execution is fast and memory consumption normal.
>
> Thanks!
>
>
​From my own experience I can tell you that Pony caching has never caused
me any problems. ​One note though, if you're using flask or any other
program that gives you memoization capabilities like `lru_cache` in Python
3 or `cached_property` werkzeug, be careful when using them since by using
them the database objects get *stuck* and are not marked for deletion later
on. Hence your cache can grow exponentially. My advice is this, pony
caching is not a problem, caching cached objects however, is.


> On 8 November 2014 13:19, Alexander Kozlovsky <
> alexander.kozlovsky at gmail.com> wrote:
>
>> Hi, Matthew
>>
>> If you retrieve these objects to render some HTML page, than I doubt that
>> you really need so many objects. I think that a typical web page requires
>> several thousands objects at most, and this amount is not big enough to
>> cause memory problems.
>>
>> > each time the inner function is called, it returns a data structure
>> full of pony objects (but a lot less pony objects than the function
>> internally used)
>>
>> Ideally you function should retrieve only the objects which are actually
>> required to render the Jinja template. If the function retrieves much more
>> objects, but then uses only small fraction of them, then it probably
>> written inefficiently. You should do object filtering not in memory, but in
>> the initial database query.
>>
>> If you can show me the database query then I probably can suggest a
>> better way to do filtering, so only required objects will be materialized
>> in memory.
>>
>> Regards,
>> Alexander
>>
>>
>> On Fri, Nov 7, 2014 at 5:40 PM, Matthew Bell <matthewrobertbell at gmail.com
>> > wrote:
>>
>>> I think the problem is that each time the inner function is called, it
>>> returns a data  structure full of pony objects (but a lot less pony objects
>>> than the function internally used). I could pull the attributes of the
>>> objects that I need (to inject into a Jinja template, but not for a
>>> webapp), but that seems hacky. I could pickle / just return the IDs of the
>>> objects, but that also seems hacky.
>>>
>>> The relevant line in the jinja template is like:
>>>
>>> {% for keyword in keywords %}
>>>    {{ keyword.expensive_method() }}
>>>
>>> Do you see any solutions?
>>>
>>> Thanks
>>>
>>> On 6 November 2014 13:05, Alexander Kozlovsky <
>>> alexander.kozlovsky at gmail.com> wrote:
>>>
>>>> Hi Matthew,
>>>>
>>>> > Does Pony's in-memory cache have a size limit?
>>>>
>>>> Currently no, but we can think about it. It may be not so easy to
>>>> implement, because objects in cache have cross-relations.
>>>> May be we can add some method like db_session.forget(MyEntity) to clear
>>>> all unmodified entities of this type from the cache. Can this help in your
>>>> case?
>>>>
>>>> >  I have a function which does a lot of queries within one transaction
>>>>
>>>> Are you sure that in your case all those queries must be done in the
>>>> same transaction? If not, then wrap different queries in different
>>>> db_sessions, and old objects should be cleared automatically.
>>>>
>>>>
>>>> On Thu, Nov 6, 2014 at 3:43 PM, Matthew Bell <
>>>> matthewrobertbell at gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Does Pony's in-memory cache have a size limit? I have a function which
>>>>> does a lot of queries within one transaction, dereferencing > 99% of the
>>>>> pony objects, but the memory used by the process grows and grows and the
>>>>> function goes through its loops.
>>>>>
>>>>> Psuedo code:
>>>>>
>>>>> results = []
>>>>> for i in range(5):
>>>>>   results.append(some_function_which_queries_lots(i))
>>>>>
>>>>> Thanks!
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Matthew Bell
>>>>>
>>>>> _______________________________________________
>>>>> ponyorm-list mailing list
>>>>> ponyorm-list at ponyorm.com
>>>>> /ponyorm-list
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> ponyorm-list mailing list
>>>> ponyorm-list at ponyorm.com
>>>> /ponyorm-list
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Matthew Bell
>>>
>>> _______________________________________________
>>> ponyorm-list mailing list
>>> ponyorm-list at ponyorm.com
>>> /ponyorm-list
>>>
>>>
>>
>> _______________________________________________
>> ponyorm-list mailing list
>> ponyorm-list at ponyorm.com
>> /ponyorm-list
>>
>>
>
>
> --
> Regards,
>
> Matthew Bell
>
> _______________________________________________
> ponyorm-list mailing list
> ponyorm-list at ponyorm.com
> /ponyorm-list
>
>


-- 
Kind Regards,

Quazi Nafiul Islam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </ponyorm-list/attachments/20141109/356a54aa/attachment-0001.html>