The following test cases are used to test, validate and/or certify PaperCut copier/device integration. The use-cases and tests apply to both PaperCut in-house developed copier control solutions (such as embedded solutions) as well as solutions developed by 3rd parties using PaperCut’s Copier Control Terminal API.
3rd party implementations must complete these test cases before being certified by PaperCut. The 3rd party should complete the tests recording the results. Where the implementation does not comply, an explanation for the non-compliance must be provided. The testing procedure below is thorough and detailed. Testers should look at dedicating at least one full day to this process and its documentation.
Test are designed around “real world” use cases and are outlined in detail below.
Test Cases Test Case 1: Student user A student is restricted in the amount of printing and copying that can perform. This user will not have access to shared accounts. They can only charge to their personal account.
Setup Set up a student test user in PaperCut. Set the user to be “restricted”. Set the user to “Automatically charge to personal account” Set the device copy cost, e.g. $0.10 per page. Tests a) User has credit balance:
Allocate the user a small balance, e.g. $2.00. Log in as the student user. Perform some copying (or scanning/faxing if supported). Verify that the job is logged correctly in PaperCut. b) User has no balance:
Set the user’s balance to zero. Try logging in as the student. User should be denied copy access with an understandable message (insufficient funds). c) User with small balance (zero stop):
Set the user’s balance to a small value, e.g. $0.50. Log in as the user. Perform a copy that will cost more than the user’s balance. Confirm the copy job is stopped when the user’s balance is used up. NOTE: Some devices can stop the job before the job starts, others might stop the job part way through. Notes for ideal implementation:
For “restricted” users, the user’s account available balance should be displayed after successful login or throughout copier usage. Test Case 2: Teacher user A teacher has access to charge copies to two faculty accounts “Science” and “Maths”. They can also charge printing to a personal account which is unrestricted.
Setup Set up a teacher test user in PaperCut Set the teacher user to “unrestricted” Change their account selection settings to “Show the Standard Account Selection Popup” and enable “select shared account with list” only. Disable the “personal” and “shared account with code” options. Create two accounts “Science” and “Maths” and grant this user access to these accounts (through Account → Security). Create a third account “Physics” and ensure the user does not have access to this account. Tests Log in to the device as the teacher user. The user should be prompted for the account to charge their copying. They should be able to select the accounts from a list. The list should be: Science Maths Once the account is selected, perform some copies. Confirm that the job is logged correctly in PaperCut and was charged to the selected account. Repeat this test, selecting each account option in turn. Confirm jobs are logged correctly. Notes for ideal implementation:
When users select “unrestricted” accounts, the account balance is not relevant and should not be presented to the user. If something is to be displayed it should be “unrestricted”. When users select “restricted” accounts, the account balance should be displayed to the user. Zero-stop rules should apply to restricted accounts that exhaust their balance. When users have a small number of accounts, it’s preferable to minimize the number of steps. It is not appropriate to provide an “Account search” option when the list is so short. When there is a small number of accounts, it is recommended to simply display them in a selectable list format. When a “default shared account” is set for the user, the user interface should change to make this account easy to select with minimal steps/presses, i.e. it might appear at the top of the account list so it can be selected with one click. Test Case 3: Teacher user (PIN/code account selection) In this organization teachers select faculty accounts using PIN/codes and continue to use this convention at the copiers. They are also not allowed to charge copying to their personal account and must select a shared account via a code.
Setup Setup a teacher test user in PaperCut Set the user to be “unrestricted” Change their account selection settings to “Show the Standard Account Selection Popup”. Disable the “shared account with list” and “personal account” options. Enable the “shared account with code” option. Set the PIN/codes on the previously created accounts as follows: Science: set code to 1111 Maths : set code to 2222 Physics : set code to 4444 (do not explicitly grant the user access to this account) Tests Log in to the device as the teacher user. The user should be prompted to enter the account PIN/code. Select account 1111 and perform some copies Confirm that the job is logged correctly in PaperCut and was charged to Science. Repeat the preceeding two steps for account 2222. Repeat for account 3333 (invalid account). Confirm that a suitable error message is displayed. Repeat for account 4444. The user should have access to Physics when using the PIN, even without security access being granted. Test Case 4: Corporate user The corporate user has their copies tracked, but they are not restricted from copying in any way (tracking/auditing only). They do not have access to charge to Shared Accounts. All jobs are automatically charged/allocated to their personal account.
Setup Set up a corporate user in PaperCut (e.g. corporateuser) Set the user to “unrestricted” Set their “Account Selection” settings to “Automatically charge to personal account” Tests Log in to the device as the corporate user. After login, the copier should be enabled for use. Perform some copying. Confirm that the job is logged correctly in PaperCut. Notes for ideal implementation:
In this scenario the objective is to grant the user copier/device access with as minimal user interaction as possible. The aim is to keep user steps to a minimum. Test Case 5: Professional user (Engineering Company) An engineering company allocates all printing and copying costs to client and project accounts. With over 5000 accounts to choose from they need to find the account efficiently via keyword search (search and review). For example, list all accounts and sub-accounts for “acme inc”.
Setup Configure the “[Template Account]” to grant access to the “[All Users]” group. This ensures all users have access to all new accounts created. Use the bulk account import feature to import 5000 shared accounts. TIP: Use Excel to create a spreadsheet with 5000 accounts named: account0001, account0002, account0003, etc. Then save the spreadsheet as a tab-delimited file to upload into PaperCut. See https://www.papercut.com/products/ng/manual/ch-shared-accounts-import-update.html for information on import file format and procedure. Set up the professional user’s “Account Selection” settings to be “Show the Advanced Account Selection popup” and disable the “Allow to charge to their personal account” option. Tests Log in to the device as the professional user The user should be prompted for the account to charge their copying. They must have: the option to select account by PIN/code; the option to select accounts from a list; and importantly the option to search for the account by name. Select the “Search” option and try the following searches: Search for “111”. This should return about 14 accounts if the accounts were created as above. Search for “1234”. This should return a single account “account1234”. Search for “acc”. This will return all 5000 accounts. The device should handle this gracefully by paging through the results (not retrieving all results at once). Select an account and perform copying. Confirm that the job is logged correctly in PaperCut and was charged to the selected account. Notes for ideal implementation:
When a user has access to many accounts it is of little use to display a very long list. The user should first be presented with the option to choose how to search for an account (e.g. by PIN/code by name, etc.). When performing the account searches, a reasonable amount of account results should be requested (e.g. only 100). If the user pages through all these results the device can then request additional results. The user should be offered a method to scroll through the list of accounts. When a “default shared account” is set for the user, the user interface should change to make this account easy to select with minimal steps/presses, i.e. it may appear as an option in the same screen as where the account search is provided. Searching for “ccoun” should also return all 5000 accounts. Test Case 6: User with New Card Users use swipe cards for authentication. The user is issued a new card and they should be able to associate the card with their user account at the device (self association).
Setup Remove any previous card number from your test user in PaperCut. Do this by selecting the user and clearing the “Card/Identity Number” field. Make sure that the test card number is not associated with any other users. In PaperCut configure the device as follows: Enable card authentication. Enable the “Enable self-association with existing user accounts” option under card authentication. Tests Attempt to log in by swiping the card The device should inform the user that the card is not known to the system, and ask if they wish to associate the card with their account (e.g. Yes/No). If the user proceeds the user should be prompted for their username/password. Enter a valid username and password. The device should indicate that association was successful. Review the contents of the Card/Identity Numbers field in the PaperCut admin console and check that the card number has been added. The device should then take them back to the initial login screen to re-swipe the card. Repeat these tests and ensure an invalid username/password does not allow association and feedback is appropriate. Notes for ideal implementation:
When a card is not known to the user the user should be prompted whether they would like to associate a card. This should not happen without the user accepting this step. Test Case 7: Secure Print Release An organization uses secure print release to hold jobs in the print queue until the user releases the jobs at the device. Note: This feature also applies to find-me printing (a.k.a. Pull printing).
Setup In PaperCut, select a printer and enable “Enable hold/release queue”. Select the device and enable “Enable print release”. Under “Displays jobs for release from the selected queues”, select the print queue configured above, and ensure “Automatically release user’s jobs upon login” is disabled. Print about 12 test jobs (to fill at least two screens of the held jobs list on the device) and confirm the jobs are listed in “Printers → Jobs Pending Release”. All jobs listed here will be ready for release on the device. Also print a few jobs as another user. Tests Log in to the device as the first user (i.e., user that printed the larger number of jobs). Select Release Print Jobs. A list of held jobs is displayed. This should contain only the first user’s jobs, not the jobs of the other user. Scroll/page through the list and check that no jobs are missing. Release some jobs and cancel others. Released jobs should be printed and logged in PaperCut. Cancelled jobs should be logged in PaperCut and not printed. The ideal implementation does depend on the amount of space and GUI options available on a given device.
Notes for ideal implementation:
When giving the user the option to select between copying and print release it is recommended to display the number of currently held jobs to the user, e.g. “Print release (3 jobs)”. Users should have the option to both print and cancel jobs awaiting release. There is a lot of information about the jobs being held (e.g. doc name, user, client machine, time, pages, cost, etc.). Ideally all these attributes would be visible to the user, but on constrained devices this may not be possible. The most important attribute to display for print release is the document name. Test Case 8 : Print Release (permissions to release any print jobs) Devices have an option that allow the logged in user to release any print jobs, regardless of owner.
Setup Prepare Test Case 7, with the additional Setup step of using the ‘Users have permission to:’ drop down box to select ‘release any print jobs (charged to their account)’. Tests Repeat the Tests from Test Case 7 this time ensuring you see all print jobs for all users. Test Case 9: Secure Print Release (auto release enabled) An organization uses secure print release to hold jobs in the print queue. When the user logs into the device they want all jobs to be released automatically upon successful login.
Setup Setup PaperCut as per Test Case 7: Secure Print Release. In the device configuration enable the option “Automatically release user’s jobs upon login”. This can be found under the device’s print release settings. Tests Log in to the device as the user that printed the jobs waiting for release. The login will trigger the jobs to be release. If jobs have been released the user should be informed by a brief message. The device should then take the user through the standard copier process and not display an option to copy/release because the user’s jobs will already be released. Notes for ideal implementation:
The message displayed on login if jobs are released should ideally indicate the number of jobs released. If multiple jobs are held for a user but not there is not enough credit to release all the jobs, user should be given an option to select the job(s) they want to release. Test Case 10: Lost card (alternate login methods) User in this scenario normally authenticate with cards, however users should also be able to log in with alternate login methods. This is useful when user has misplaced their card (e.g. left their card at home). An alternate login method, although less convenient, will ensure the user still has copying and secure print release access.
Setup In PaperCut enable both the “swipe card” and “username and password” authentication. Make sure your test user has the card number set to the number of your test card. Tests When the user approaches the device they should be able to log in with either card or username/password. Test that the user can successfully log in with a card. Test that the user can successfully log in with a username and password. Notes for ideal implementation:
The user should be able to log in with a card without pressing any keys on the device. i.e. they should not have to first select “card authentication”, nor have to push “Enter/OK” after a card swipe. The interface should guide the user to the options available. The interface will depend on the capabilities of the device. On a limited display device the screen might show “Swipe card or press X to enter username/password”. On a graphic/touch screen, the alternate login options may be presented as buttons. The interface should where possible support all authentication methods: Card Swipe Username and password ID and PIN Password and PIN fields should be masked when entering values. When entering an ID the “Mask identity number” (configured on the server) setting must be honoured. Test Case 11: Two Factor Authentication To prevent unauthorized card use when a card is stolen or lost, organizations may chose to implement two factor authentication (something you have plus something you know). In PaperCut two factor authentication is implemented via a numeric card PIN.
Setup In PaperCut, enable the “Swipe card” authentication for the device. Enable the “Require PIN” option under the “Swipe Card” option. Setup the test user: Ensure the user has credit/balance on their account. Ensure a test card number is entered in the test user’s “Card/Identity Number” field Ensure the the “Card/ID PIN” field is clear/empty. This will cause the device to request a new PIN on first use. Tests Test for user without a PIN (define new PIN on first-use):
Swipe the test card at the device. The user should be prompted to enter a new PIN. They should also be asked to confirm the new PIN to ensure it was correctly entered. The user should be informed that the new PIN is assigned and device access granted. Test for user with a PIN:
Now that the user has been assigned a PIN (in the previous test), swipe the card at the device again. The device should prompt for the PIN. Enter a correct PIN and ensure device use is granted. Enter an invalid PIN and ensure device access is denied. Notes for ideal implementation:
Where possible the user interface should adapt based on whether a PIN is required. Message should appear on the screen helping the user understand why they have been requested for a PIN. Suggested text: “Please enter a new PIN to associate with your card.” If using “ID” authentication the user interface should only request a PIN if one is required. In the GUI, this might mean there is no PIN field if a PIN is not required. Test Case 12: Account tracking only (auto login) A company wants to charge printing to accounts, but is not interested in tracking copying at the individual user level. It is more important for them to speed up access to the copier by not requiring that users log in.
Setup Create a test user called “copieruser”. In PaperCut under the device’s authentication option, enable the “Anonymous (No login required)” option. Enter the name “copieruser”. All jobs will be tracked against this user. Under Users, select “copieruser”. Change the Account Selection options to “Show the standard account selection popup”. Enable the “select shared accounts (from list)” option and disable all other options. Ensure copier user has access to at least two Shared Accounts (e.g. Shared Accounts → Account Details –> Security). Tests User with account selection:
The device should no longer prompt for login. The user should be presented with the account selection options. Select an account and perform copying Verify that the job was logged correctly. The job should be charged to the selected account and associated with the “auto login” user. No account selection:
Change the account selection option for “copieruser” to “Automatically charge to personal account” When using the device there should be no login or account selection to enable the copier. The device may request the user to press a button to proceed and enable the copier). Notes for ideal implementation:
If there is account selection enabled, then the device should send the user straight to the account selection screen without any user interaction. It may for example be appropriate to simply display a “start” button rather than offer a login interface. If there is no account selection (e.g. automatically charge to personal account) the device may display an initial prompt for the user to enable the copier, e.g. “Press to begin”. Test Case 13: General Copy Job Attribute Tests Conduct some general copy job attribute tests making sure the job is correctly recorded in the device’s job log (Device → [select device] → Job Log). At a minimum test the following job settings (if supported by the device):
Letter A4 Legal 11×17 A3 Duplex Black & White/Grayscale vs. Color Copy zooming (making sure the size recorded is the output size rather than the scan size) Test Case 14: ‘Enable hold/release queue – disabled’ When an embedded application is installed on the MFD and the queue that services the device in PaperCut is not configured as a hold/release queue, prints should emerge from the device without any user intervention at the device itself.
Setup Successfully complete Test Case 7 Uncheck the ‘Enable hold/release queue’ option in PaperCut for this print queue Tests Print some test jobs to the queue and confirm their pages print without any additional intervention. Test Case 15: Fax and Scan Tests If the integration is designed to target/track Fax and Scan usage, the above test cases (at least the major ones) should be repeated for these event types.
Verify that scan tracking works for all possible storage cases:
Scan to email Scan to network file store Scan to device document store Scan to locally connected USB media Test Case 16: Exception, Security and General Tests The following tests cover security, exception and general tests. The focus is on tricky corner-cases that may show up bugs or unexpected behavior in real-life environments. Please record the result of each section below and allow adequate time to complete all tests. These tests are some of the most important in the series and are often the ones that demonstrate problems with initial implementations.
Device naming The device must provide a way to set a device name. This will typically be configured along with the server IP address. The device name is used to identify the device in PaperCut. Invalid server When the device is configured to point to an invalid server IP address (e.g. an incorrect IP address). A suitable error message should display. (e.g. Communication Problem: Unable to connect to server) PaperCut Application Server restart test While device is displaying idle/login screen restart the PaperCut application server Confirm you can log in after the restart is completed. Disconnect network cable (while trying login) While at login screen, disconnect network cable. Try logging in, etc. Confirm that it handles the error gracefully, i.e. displays a reasonable message. Reconnect network cable. Try logging in again and confirm that all works as expected. Disconnect network cable (leave idle) While at login screen, disconnect network cable. Leave device idle for 10 minutes (do not log in). Reconnect network cable. Try logging in again and confirm that copy tracking works as expected (e.g. perform a test job). Disconnect network cable (prior to copy job) While at login screen, disconnect network cable. Perform a copy job Leave a minute Reconnect network cable. Verify that after a few minutes the copy job is tracked in PaperCut. No network at power on Turn off device. Disconnect network cable. Restart the device. Confirm that the application displays a reasonable message when network disconnected. Reconnect the network cable. Confirm the device connects and you can log in. Stop PaperCut Application Server during copying Start a large copy job (suggested 100 pages). While in progress stop the PaperCut application server (e.g. Services → PaperCut Application Server → Stop). Wait for job to complete, etc. Restart the Application Server a few minutes later, and confirm that the job is logged correctly. Hard power-off during copy (NOTE: don’t do this too often may be not good for device)
Start a large copy job. After about 10 pages, hard power off the device (e.g. power off at the power point). Note the number of pages actually produced. Restart the device. Confirm that the pages are counted correctly (within reason as some may still be half stuck in the device). Invalid license When the device is configured to point to a PaperCut MF server with: no valid device licenses (when using a permanent license), the device should display a suitable message indicating a license is required (i.e. the message returned by the server’s registerDevice call). no valid device entitlement (when using a subscription), the device should display a suitable message indicating it’s in an unlicensed state. See Managing entitlements (Identifying devices that are not licensed) for more information. Device automatic reconfiguration Make a price configuration change on the device in the PaperCut administration console. Verify that the device automatically detects this change within 60 seconds.
Change other configuration options that are statefully stored on the device (e.g. price line settings) and verify they are detected within 60 seconds.
(Note: It is acceptable for the device to trigger off a “reboot” to refresh its settings. This is often the simplest way to implement configuration change detection.)
Large copy job Perform a 100 page copy job … allow it to complete and confirm it’s logged correctly. Start a 500 page copy job …. cancel it after about 10 pages have been copied and confirm it’s logged correctly. Long Shared Account name The device must handle long Shared Account names. Add a new account called: “This is a very long account name and should not affect screen layout or usability”. Verify that account can be selected by the user and it does not disrupt any screen layout (truncating names with “…” may be acceptable). Long Print Job names The device must handle the secure release of jobs with long names. Try printing a document from such as a MS Word document with a long file name, or a web page with a very long URL. Verify that the print job can be released by a user and the long name not disrupt any screen layout (truncating names with “…” may be acceptable). Names with special characters The device should handle printer, document, account, user and workstation names with special characters. Characters to check include: < > & ; @ ' " and non-Latin characters as appropriate. Take time to check each area. Test with a different country display setting. Go into Options → Display Options → Location and change the language to one of the supported languages which is not English (and preferably that uses non-English characters - e.g. Chinese, German, etc.). Check on the device that the characters are displayed properly. Large Shared Account Lists The device must be able to handle listing of large account lists. Some PaperCut customers have over 100,000 shared accounts (e.g. client accounts). Test and make sure the device performs acceptably with 100,000 shared accounts. (Tip: Many accounts can be created quickly using MS Excel and the batch account import process.) Card reader stability (for removable USB card readers) Plug in card-reader and confirm it’s working (e.g. log in). Unplug the USB card-reader. Leave for a couple of minutes. Reconnect the card-reader. Test whether it works or confirm a reasonable error message is displayed. Power saving mode test If the device runs, or connects to an MFD/MFP ensure that the integration does not stop the MFD from entering any of the manufacturer’s designed power saving modes. Make sure when the device enters a power saving mode, it still appears as connected/idle in PaperCut’s device status. Ensure functionality returns after the MFD is woken from power saving mode(s). Power-off idle test Turn the MFD/Device off (full power off at the power point). Wait 10 minutes. Make sure the device enters an “Offline” state in the PaperCut Admin Console’s device status. (Note: It should not enter an “Error” state.) Overnight idle test Confirm the device standby/power-save timeout is enabled. Plug in card-reader and confirm it’s working (e.g. log in). Leave device on overnight (or for a period longer than the standby timeout) while connected to PaperCut. Some devices have various sleep phases. Suggest leaving overnight for a complete test. In the morning, test log in (including the card reader) and ensure everything works as expected. Review memory consumption on the PaperCut server, and the device, between the start and end of the test to identify possible memory leaks. Scan exception testing Load the document feeder with a large document and start scanning. Cancel scanning via cancel button and log out. Ensure job tracks/records as expected. Fax exception testing Verify that a Fax job with an invalid phone number does not charge Verify that a delayed Fax job eventually logs, e.g. unplug phone cable for 10 minutes, reconnect and ensure the job is logged when it finally completes. Verify Installation Process Install the solution from scratch using the supplied documentation Note any issues or problems Cross-platform Testing If your solution/device communicates with the PaperCut server through any other protocol, port, or method other than those recommended by PaperCut, then the solution should be tested on all platforms. Verify encrypted traffic Install Wireshark ( http://www.wireshark.org/ ) or equivalent on the system running the PaperCut Application Server Enable logging of all traffic between the server and the test devices Conduct a card swipe login. Verify that the card number was not passed in clear text (e.g. search packet content using Wireshark search). Conduct a username and password login. Verify password was not passed in the clear. Support for multiple currencies and multiple languages (if available) Does the device allow the locale to be set in the PaperCut administration interface (Options → Display Options) or can the locale be set via the solution interface? Do these supported locals support all the regions in which the solution will be marketed (refer to Go To Market plan)? Does the solution display the correct currency symbol when the locale is changed? (e.g. “€”) Do the messages displayed to the user change to the appropriate language when the solution locale is changed Does the supplied documentation explain how to set the locale correctly Security Review All certified solutions that are intended to be deployed in an education (or other) environment must be audited by PaperCut. The following list provides details on the most common functions and areas of analysis that may cause negative feedback:
No Zero Stop: Inability to stop a copy or scan job while in progress (say due to an account running out of quota/credit) Information-loss due to power cycle or network disruption: The solution should have the ability to persist and remember job details and progress between restarts, power-cycles and network interruptions. No job information should be lost as a result of a power-cut or network loss (either inadvertent or malicious) at any stage in any job operation. Limited Feature Control: Devices should offer the ability to limit user access/options by: Functionality (e.g. scan - no copy) Color vs. Grayscale (e.g. grayscale only - no color) Limited Secure Protocol Support: Devices should support modern secure communication protocols (e.g. HTTPS/TLS) for all user-authentication related communication between the device and the server. Known Device Security Issue: PaperCut’s in-house development team will check devices for known security issues. Issues found in the past include: options to reset hardware back to factory default without requesting admin passwords insecure system boot allowing application reset/bypass All issues discovered will be reported privately to the vendor and PaperCut will work with the vendor to action as appropriate.
Limited Production Maturity: Any new solution with no known issues at the time of release will reviewed again after a 12 month security-issue free period. Large Scan Jobs If the integration is designed to target Scan usage, testing should verify that large scan jobs complete as expected.
Perform a large scan job, in excess of 50 two-sided (duplex) pages. Allow it to complete and confirm it’s logged correctly. Start a 100 page scan job. Cancel it after about 10 pages have been scanned and the job should not be logged.
Paused Scan Jobs If the device is capable of pausing a scan job, testing should verify that logging out of the device cancels the scan job as expected when using a logout button provided on the device, and leaves the device in a good state for other users after logout. This test requires two users, User A and User B, both with sufficient funds and permissions to use integrated scanning.
Log in to the device as User A. Perform a multi page scan job, and pause the job before the completion of the scan job Log out of the device using the button provided on the device. Verify that the scan job is logged as cancelled in job logs for User B Log in as User B, and verify that Integrated Scanning is able to successfully complete a new scan job for the User B Log out of the device using the button provided on the device Verify that the scan job is logged successfully in job logs for User B
Comments
0 comments
Please sign in to leave a comment.