Memory consumption/leaks

Jul 7, 2010 at 7:17 PM

I've noticed when loading large data sets (25K + records) the memory consumption gets balooned very fast to about 0.5 GB from mere 50MB, while the actual View Model collection is only 12MB.
The Ants profiler tells me that the rest of the memory is used by Action, List<Dependent>, and some other things from UpdateControls library.
And then, when view model is finally discarded I have to go through each individual _indMyField variable and set it to null (there is no Dispose method). Otherwise GC never reclaims memory back.

Is there a way to minimize memory consumption and/or have Dispose method for Independent class which will break all the links between objects.

Thanks!

 

Jul 7, 2010 at 8:01 PM

Here is a link to a screen shot.

 

Coordinator
Jul 7, 2010 at 8:56 PM
Edited Jul 7, 2010 at 8:58 PM
Update Controls keeps two-way links between the Precedent and the Dependent. Precedent is the base class for both Independent and Dependent, so it can handle indirect dependencies. If you have a dependent property that accesses a member of every item in a large collection, that might become a large number of links. For example, if you are sorting in your view model, you might have this:
public IEnumerable<PersonViewModel> People
{
  get
  {
    return _model.People
      .OrderBy(p => p.LastName)
      .Select(p => new PersonViewModel(p));
  }
}
The People property will be dependent upon the LastName property of every person in your model. That one Dependent will have a large List<Precedent>, and each Precedent will have a List<Dependent> with one item. You can remove the sorting from the view model and instead let something like PagedCollectionView handle it. You might even want to use an ObservableCollection for that one property, which will tell Update Controls to get out of the way. If the dependency isn't obvious, you can build against the source code rather than the compiled assembly. Set a breakpoint in Dependent.AddPrecedent.
Coordinator
Jul 7, 2010 at 9:04 PM
Minor correction: Precedent is not a base class. It is used via composition by both Independent and Dependent.
Neither Independent nor Dependent implement IDisposable, but Dependent does have a Dispose() method that tears down dependencies. For Independents, OnSet() has the same effect. Just be sure that the dependent view has closed, or else it will just set them back up again.
Jul 7, 2010 at 9:57 PM

Thanks Michael.

I have to digest what you have told me. It makes a lot of sense. Since I am using Independent properties as filters for collections.

And the more filters I use the larger the memory footprint.

Like so:

IQueryable<EntityViewModelBase> list = (from e in _invoices
            where e != null 
             && (IsOpen == true) 
             && ((String.IsNullOrEmpty(PONoFilter))
                     || (!String.IsNullOrEmpty(PONoFilter) && e.PONo.Contains(PONoFilter)))
             && ((String.IsNullOrEmpty(InvoiceNoFilter))
                     || (!String.IsNullOrEmpty(InvoiceNoFilter) && e.InvoiceNo.Contains(InvoiceNoFilter)))
             && ((String.IsNullOrEmpty(NotesFilter))
                     || (!String.IsNullOrEmpty(NotesFilter) && e.Notes.Contains(NotesFilter)))
             && ((!ApprovedFilter.HasValue)
                     || (ApprovedFilter.HasValue && ApprovedFilter == e.Approved))
             && ((!RejectedFilter.HasValue)
                     || (RejectedFilter.HasValue && RejectedFilter == e.Rejected))
             && ((!ReportedFilter.HasValue)
                     || (ReportedFilter.HasValue && ReportedFilter == e.Reported))
             && ((!PrintedFilter.HasValue)
                     || (PrintedFilter.HasValue && PrintedFilter == e.Printed)) 
             && ((!DateApprovedFrom.HasValue)
                     || (DateApprovedFrom.HasValue && e.DateApproved.HasValue && e.DateApproved >= DateApprovedFrom))
             && ((!DateApprovedTo.HasValue)
                     || (DateApprovedTo.HasValue && e.DateApproved.HasValue && e.DateApproved <= DateApprovedTo))
             && ((!DateSubmittedFrom.HasValue)
                     || (DateSubmittedFrom.HasValue && e.DateSubmitted.HasValue && e.DateSubmitted >= DateSubmittedFrom))
             && ((!DateSubmittedTo.HasValue)
                     || (DateSubmittedTo.HasValue && e.DateSubmitted.HasValue && e.DateSubmitted <= DateSubmittedTo))
             && ((!DateInvoicedFrom.HasValue)
                     || (DateInvoicedFrom.HasValue && e.DateInvoiced.HasValue && e.DateInvoiced >= DateInvoicedFrom))
             && ((!DateInvoicedTo.HasValue)
                     || (DateInvoicedTo.HasValue && e.DateInvoiced.HasValue && e.DateInvoiced <= DateInvoicedTo))
             && ((!DatePOFrom.HasValue)
                     || (DatePOFrom.HasValue && e.DatePO.HasValue && e.DatePO >= DatePOFrom))
             && ((!DatePOTo.HasValue)
                     || (DatePOTo.HasValue && e.DatePO.HasValue && e.DatePO <= DatePOTo))
             && ((!DateRejectedFrom.HasValue)
                     || (DateRejectedFrom.HasValue && e.DateRejected.HasValue && e.DateRejected >= DateRejectedFrom))
             && ((!DateRejectedTo.HasValue)
                     || (DateRejectedTo.HasValue && e.DateRejected.HasValue && e.DateRejected <= DateRejectedTo))
             && ((serviceVendorId == 0)
                     || (serviceVendorId > 0 && e.ServiceVendorId == serviceVendorId))
            select e).ToList().Select(e=> InvoiceViewModel.Create(e, DataContext.Invoices) as EntityViewModelBase).AsQueryable()
I have to rethink this strategy.
Jul 8, 2010 at 6:28 PM
Edited Jul 8, 2010 at 6:30 PM

I changed queries to use Find command instead of dynamic filtering.

Now GC collects all those objects just fine.

Thank you very much for your comment it cleared up a lot of things for me.

Queries still use more memory than I would like, but it has gotten better in that sense. I don't want to change collection to ObservableCollection, since I am using IQueryable interface to make sure that every time propery gets refreshed the data is requeried from database. I do have some special management with regards to when requery the data. Also my datamodel doesn't use UpdateControls, I still use PropertyChange as in the original LINQ to SQL generator. My ViewModel objects are the ones using UpdateControls. Simply because all my business rules, validations and rollbacks occur in that layer and it is a whole lot easier with UpdateControls to handle refreshes.

So here is an example of how memory usage progresses from layer to layer.

Data layer (LINQ objects) collection - 1 MB
View Model collection for the same LINQ objects - 10 MB
UpdateControls wrappers over ViewModel collection - 200 MB

Aug 3, 2010 at 12:20 AM
Edited Aug 3, 2010 at 12:21 AM

Michael,

I decided not to use UpdateControls for large collections, and instead create view models (with UpdateControls) inside navigation model. This way collections are light weight with minimum events and behaviors.

I tried loading 160K records with about 30+ fields and I get OutOfMemoryException. That's around 4.8 mil instances of Independent class. When app memory consumption reaches 1.4 GB the app crashes.

I tried everything I know so far. May be there is another way?

Thanks,

Ivan.

 

Coordinator
Aug 5, 2010 at 4:56 PM

That sounds like the right decision. You don't want large collections in memory, let alone to add dependency tracking to them.

You can bind a data grid to a paged collection view, and let it manage paging, sorting, and caching. You won't get the advantages of a view model layer when you do this, so just show actual data columns in the data grid.

When the user selects an item from the grid, you can load the record into a business object with Independent sentries. This would back the view model for the details pane. You'll have to handle the right events to save changes to this business object back to the database.

Another alternative is to switch your UI to a search metaphor. Instead of showing the user thousands of rows, guide them through a search. Only bring at most 100 results back from any particular query, and ask the user to narrow it further. Maybe you can show them tags with counts to help them apply new filters. This would all be driven from the database, so again Update Controls only enters the picture when you have screen-sized data.

In either case, the answer is to use in-memory business objects with Independent sentries only for data that the user can reasonably navigate. For large data sets, keep it in the database.

Aug 6, 2010 at 6:33 PM

Thank you, Michael!

I can't seem to find PagedCollectionView for WPF. I've been thinking about implementing paging myself. But I don't think I can make it as good as the one for Silverlight.

I do have search filters, but users still require to bring as much data as they requested from DB. Which could be 100K+ records, because they have the ability to copy and paste it into Excel and further analyze data in there.

So I'll have to wrap objects into view model with Independent sentries only when needed, just like you said.